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Dear Friends of the WICFIE Mental Hea Ith Program: 

Founded in 1953, WICFIE and its 16 member states and territories work collaboratively to expand educational 
access and excellence for all citizens of the West. By promoting innovation, cooperation, resource sharing, 
and sound public policy, WICFIE strengthens higher education's contributions to the region's social, economic, 
and civic life. In 1955, WICFIE established its Mental Health Program, and in 2015 we are celebrating our 60th 
Anniversary of service to the West. 

People often ask what a mental health program is doing in a higher education organization. The answer to that 
question can be found in our archives, of discussions related to our founding: "a key element of access and 
success in higher education is a healthy mind." For the past 60 years, the Mental Hea Ith Program has served the 
West in partnership with our state and territorial partners to improve behavioral health services and to ensure a 
high-quality behavioral health workforce. 

Key to success in our mission is a focus upon supporting our partners in assessing and analyzing data to better 
understand need, as a means to inform and guide sound policy and practice decision making. Accountability 
requires sound data. Evidence based treatment requires sound data. Quality healthcare requires sound data. 
Across our 60 years of service to the West, the WICFIE Mental Hea Ith Program has promoted data driven practice 
and policy formation. 

It is a pleasure, as the first event of our 60th Anniversary Celebration year, to offer the publication of Human 
Services Program Evaluation - How to improve your accountability and program effectiveness, written by Thomas 
Barrett, Ph.D. and James Sorensen, Ph.D., both longtime friends of the WICFIE Mental Health Program, and 
leaders in the area of Program Evaluation, and data driven decision support. I hope you find this book to be of 
great utility to your work, as I know it is intended to be. The first step in care is to seek to understand. 

Respectfully, 


Dennis F. Mohatt 

Vice President for Behavioral Hea Ith 


www.wiche.edu 
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Introduction 

Why is it important to measure the impact of human service programs? 

The term "outcome evaluation" has become one of the most popular terms among human service 
providers and those whose jobs it is to evaluate the impact of human service programs. State and federal 
legislators, state and federal officials, and private accrediting organizations rarely finish a day without 
bemoaning the lack of appropriate outcome evaluation data. Almost everyone in the human service field 
would agree that there is not sufficient information about whether or not most human service programs 
are doing what they are supposed to be doing. 

Over thirty-five years ago there was recognition of the need for good program evaluation and good 
outcome evaluation for human service programs. 

As the demand for comprehensive program evaluation in CMHCs increases, more emphasis is 
being placed on the importance of each program's accountability for the delivery of effective 
treatment services. Client outcome evaluation techniques are the primary tools available for 
assessing the effectiveness of treatment services, and their basic purpose is to measure the 
impact of treatment services on the problems and lives of the clients who receive the treatment. 
(NIMH, 1976, p. 201) 

If there was this recognition thirty-five years ago, why don't we have better outcome information on 
human service programs today, and why isn't there better agreement on what are the appropriate 
outcomes and how to use them to evaluate programs? 

While a detailed historical perspective would fill the pages of a book by itself, some historical analysis is 
necessary here. Curiosity alone would be sufficient motivation for most of us to delve into some analysis 
of this dilemma. More importantly, the problems associated with the delay in the advancement of the 
field of program evaluation must be addressed or - as historians would warn - we are doomed to repeat 
the mistakes of the past. 

In 1980 when Tom first began teaching a graduate level class in program evaluation, he asked the 
students to give him their perspective on why there is so much resistance to program evaluation of 
human service programs. The students were anxious to answer because many of them were being 
forced to take this program evaluation course because it was (and still is) a required course for the Psy.D. 
degree. The answers focused on the difficulty of evaluating human service programs and many said that 
it was impossible to accurately evaluate therapy because it does not lend itself to objective evaluation ! ! ! 

Over the years (Tom taught this same required class for the time period between 1980 through 2003 and 
again from 2010 through the current time) the responses have begun to change. He no longer hears the 
response that it is not possible to evaluate services objectively, although he still hears that it is difficult 
to evaluate services. Clearly, the attitudes of graduate students have changed over the years. Graduate 
students now recognize the importance of measuring client outcomes. If there is increasing recognition 
of this importance, why aren't there better outcome evaluation systems today? 
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Why outcome evaluation systems aren’t more sophisticated? 

In the last twenty years there has been a proliferation of outcome instruments. In the public sector 
alone, there are over a hundred instruments in use to evaluate the impact of state human service 
programs. Most states, many providers and most accrediting bodies have taken on the challenge of 
developing better program evaluation because there is universal agreement that better evaluation 
systems are needed. This multiplicity of approaches may have been appropriate for the early stages of 
development, but eventually some standardization is necessary. Until recently, everyone has approached 
the challenge of evaluating human service programs a little differently. 

Should evaluation focus on process or outcomes? 

Accrediting bodies and many providers have chosen to focus quality improvements efforts at whether or 
not standard practices have been utilized. Unfortunately, this has resulted in focusing attention on the 
processes used in providing human services with little attention devoted to evaluating the outcomes. 

For example, the National Quality Forum (NQF) in a release in 2012 endorsed behavioral health mental 
health measures that included: 

• 1937: Follow-Up After FHospitalization for Schizophrenia (7- and 30-day) (NCQA) 

• 0576: Follow-Up After FHospitalization for Mental Illness (NCQA) 

(National Quality Forum Website, accessed on June 17, 2014). These measures evaluate whether 
or not patients are seen within certain time periods after hospitalization - 7 and 30 days. There 
is nothing inherently wrong with evaluating the procedures used in providing service and there is 
nothing inherently wrong with these evaluation measures. It makes imminent sense to suggest that 
seeing a patient within thirty days after discharge from an inpatient setting makes good sense. In fact, 
improvements in service program are dependent on evaluating whether or not the procedures used are 
the best or better than some other way of doing things. Flowever, there is a problem with starting the 
evaluation with an analysis of procedures. 

The problem is - what criteria do you use to determine whether or not the procedures are effective? 
Flow should one select the best measures? What is the basis for choosing which procedures should be 
measured as indicators of quality services? Since you must have some criteria, you are forced to choose 
from these options: 

1. Procedures are good if everyone agrees that they are good procedures. 

2. Procedures are good if respected professionals have used them for a very long time. 

3. Procedures are good if they produce the intended outcomes. 

Given these options, there is really only one good choice- #3. Options one and two may be useful for 
some purposes but they have the very real potential disadvantage of supporting procedures that don't 
work! ! Consequently, procedural measures should be utilized in the following instances: 

1. When good outcomes measures are not available; 

2. When the relationship between the procedures and good outcomes has been established. 


2 


Human Services Program Evaluation 

How to improve your accountability and program effectiveness 


Introduction 


For these reasons outcome evaluation systems should almost 
always have process as well as result measures. As evaluation 
systems become more sophisticated, measurements should be 
focused on outcomes and those process measures that have an 
established relationship with good outcomes. 

What are appropriate outcomes for Human Service 
Programs? 

One of the difficulties in measuring outcomes for human service 
programs is that there are no universally accepted outcomes. 

There are commonly accepted criteria in many other areas. 

For example, in the medical field, there are clear criteria for 
determining whether or not a heart transplant has been 
successful. The lack of enthusiasm for identifying outcomes has 
been a genuine disservice for human service programs. Failure to 
develop agreement on outcomes has damaged human services 
programs and has resulted in reduced public understanding and 
even more importantly public confidence of human services. The 
ultimate penalty is that many people who need these services go 
without benefit of these services. 

What, then, is the answer to the original question - Why 
haven't we made more progress? This book will clarify some 
of the issues in answering this question and will propose some 
solutions to this dilemma. 

What is different about evaluating human service 
programs internationally. Don’t program evaluation 
principles have universal application? 

Most people would agree that it is just as important to evaluate 
the impact of human service programs in low and middle 
income countries as it is in high income countries. Flowever, 
most published program evaluation comes from higher income 
countries. A quick review of Pubmed or the percentage of 
professional journals based in high income countries makes it 
clear that most of the published literature is from higher income 
countries. It is true that many of these journals (e.g. Psychiatric 
Services) publish studies from countries around the world. 

Flowever, this is clearly the minority of all published articles. 

There are many reasons for this discrepancy. The lack of resources is often one of the issues. Another 
is that there is less opportunity to do good program evaluation when there are few mental health 
programs to evaluate. For example, in many countries included in the 42 country World Health 
Organization analysis of Assessment Instrument for Mental Health Systems (WHO-AIMS) there is a very 
limited number of mental health professionals (Saxena et al, 2011). Afghanistan has a population of over 


Continuing Current Interest 
in Quality and Outcomes 

In summarizing a recent major study 
produced by Brandeis University 
and funded by the Substance 
Abuse and Mental Health Services 
Administration (SAMSHA), McCarty, 
et. al. conclude: 

"Investments in outcome 
monitoring are critical for 
assessing the quality and value 
of mental health and substance 
abuse treatments. Moreover, 
outcomes monitoring may 
help reassure stake-holders 
that effectiveness and quality 
expectations have not been 
compromised and that sufficient 
resources are available for 

service delivery Data are 

most useful to the extent that 
they are timely and provide 
insights into current operations. 
Infrastructures, therefore, 
should stress the need for real- 
time access to data and the 
active application of the data to 
quality improvement activities." 

McCarty, D., Dilonardo, J. & Argeriou, 

M. (2003), State Substance Abuse 
and Mental Health Managed Care 
Evaluation Program. Journal of 
Behavioral Health Services & Research, 
2003, 20 (1), 7-17. 
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25 million people and there were two psychiatrists in the entire country when the WHO-AIMS report 
was published in 2006 (WHO-AIMS country report Afghanistan) Also, many of these 42 countries have no 
or very few mental health services outside of the hospitals. 

Nevertheless, it is true that some program evaluation principles apply everywhere. For example, most 
funders require that some form of program evaluation for funded programs. And most funders are 
more interested in the outcomes of the services provided rather than process evaluation (Barrett et al., 
1998). In fact, the term "outcome evaluation" has become one of the most popular terms among human 
service providers and those whose jobs it is to evaluate the impact of human service programs. Within 
the United States, legislators and private organizations often bemoan the lack of appropriate outcome 
evaluation data. In the international community, NGOS and the UN organizations are struggling to find 
outcome indicators that can demonstrate the impact of programs funded through these organizations. 
Almost everyone in the human service field would agree that there is not sufficient information about 
whether or not most human service programs are doing what they are supposed to be doing. 


References 
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OK. So you want to be a clinician and you are wondering why you need to learn anything about a non- 
clinical activity like program evaluation. If you want to be a good clinician you need to know how to 
evaluate your work objectively. Program evaluation will teach you how to do that. 

Or you are an administrator. Program evaluation will provide information on how to objectively evaluate 
your programs so that they will be more effective and more efficient. And this book will show you how. 

There are other benefits to knowing about program evaluation: 

Firstly, program evaluation will make you more marketable as a clinician. Most agencies (in the US and 
around the world) value a clinician with program evaluation skills. In fact, many positions within the 
United Nations organizations (especially the World Health Organization and UNHCR) are actively looking 
for people with program evaluation skills and experience. 

Also, the need for clinicians and administrators with program evaluation skills is increasing. And the 
demand for objective information about program outcomes will likely increase over the next few years 
as funders and legislators step up their pressure to make providers accountable for good outcomes. 

The combination of clinical skills or administrative skills and good applied evaluation skills will make you 
especially marketable! 

What is program evaluation? 

Theoretically, any kind of evaluation activity of a "program" or service can be program evaluation. For 
example, talking with someone about your likes and dislikes is a program evaluation! However, good 
program evaluation strives to either present subjective information "in context" or to produce objective 
information that is based on more than on person's opinions. Your personal like and dislikes may not be 
shared by the majority of the population. That does not make your opinions invalid, but it does mean 
that your opinions should be combined with other personal opinions about likes and dislikes. In this 
way, your opinions are "in context" and are not presented as if they represent the opinions of everyone. 
Program evaluation will teach you how to use all of this information to improve program outcomes. 

Another way to define program evaluation is: a systematic determination of a program's effectiveness 
and efficiency. Effectiveness is the extent to which a program is doing what it is intended to do. Efficiency 
is the extent to which these outcomes are achieved with the minimum expenditure of time, effort and 
other resources. Another key word in this definition is "systematic". More on that later! 

When was the first program evaluation? 

Programs, people and services have been evaluated for a very long time. 

God looked at everything he had made, and he found it very good. Evening came, and morning 
followed - the sixth day. (Genesis 1:31) 

What is relatively new is the development of the discipline of program evaluation-the development of 
methodologies and principles that increase the accuracy and objectivity of information so that services 
and outcomes are maximized. 
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What are the methods that were used to evaluate programs before formal program 
evaluation? 

Before the existence of formal program evaluation in the 1960s human service programs were evaluated. 
Similarly, employees were evaluated before the existence of a formal employee performance evaluation 
system. 

In the United States and at the international level there were some methods that had been used for 
years to make these kinds of decisions. One of these methods can be best described as the Charity 
Model (Patton, 1981) where decisions about a program were made based on the sincerity and 
enthusiasm of the participants in the program. If the participants were enthusiastic, the program was 
considered a good program and funding was likely to continue. Another method is best described as the 
Pork-Barrel (Patton, 1981) philosophy where a program's value is based on the power and leverage of 
the constituency. In this methodology if the people that benefit from the program are "well connected", 
the program will likely continue to be funded. And yet another method is the highly sophisticated 
"squeaky wheel" philosophy, which supports the programs that are able to garner the most popular 
attention. While these methods are not without merit, formal program evaluation methods strive to be 
more sophisticated than these traditional methodologies. 

What were the factors that forced the development of formal program evaluation 
activities? 

It would be a great tribute to human service professionals if we could truthfully say that the force 
behind the development of formal program evaluation was a hunger for truth amongst professionals. 
Unfortunately, that was not the case. Certainly, many professionals were committed to better 
information on service programs. However, the major driving forces behind the development of formal 
program evaluation were federal legislation and pressure for accountability from funding agencies at 
various levels. 

Federal legislation came in 1975 in the form of PL94-63. This particular piece of legislation played a 
major role in shaping program evaluation activities for the next few decades. PL94-63 made a number of 
requirements of all federally funded Community Mental Health Centers. These requirements included: 

1. A sum of money equal to 2% of the previous year's operating budget had to be devoted to 
program evaluation activities. 

2. Each program was required to develop a quality assurance program for clinical services. This 
program had to include two components- utilization review and peer review. 

3. Each program had to conduct a "self-evaluation" that would include an evaluation of: 

a. Cost of operations. 

b. Patterns of use. 

c. Availability, awareness, acceptability, and accessibility. 

d. Impact of the services upon mental health of residents. 

e. Effectiveness of consultation and education services. 

f. Impact of the center on reducing inappropriate institutionalization. 

4. Each program was required to conduct a "resident's review" which would include making the 
evaluation data public, publicizing an opportunity to review services and offer comments, and 
producing a report outlining the public meeting activities. 
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While this legislation forced all federally funded mental health centers to conduct the required 
program evaluation activities, many programs conducted these evaluations reluctantly and with all the 
enthusiasm normally associated with shotgun weddings. One of the consequences of this legislation was 
the rapid development of program evaluation technology. (For example, by 1980 all Community Mental 
Health Centers in Colorado had hired a program evaluator and many centers had full blown program 
evaluation sections.) One of the negative consequences was the development of poorly integrated and 
unfocused evaluation efforts at some centers. 

Pressures for performance and outcome measures continue at the international and at the federal 
(and state) levels within the US. "States that implement managed care for publicly funded alcohol and 
drug abuse treatment services should anticipate needs to invest in the development of performance 
measures and outcome monitoring [emphasis added] so they can assess quality of care as well as cost, 
access, and utilization." (McCarty, Dilonardo & Argeriou, 2003) 

Since the authors of this book are committed to the principles of providing information that is useful 
in conducting program evaluation in the real world, it is important to be honest about the reasons why 
formal program evaluation came into existence. Failure to respond to theses forces, such as legislative 
pressure and pressure from funding organizations, will result in unresponsive program evaluation. 

What other areas could benefit from program evaluation? 

There are many performance areas that could benefit from good program evaluation- for example, 
therapist evaluations in provider organizations. Generally, therapist evaluations are based on subjective 
information about performance (Barrett and Dehaan, 1980). Employee evaluations, promotions, hiring 
and firings are often completed without objective information. A formal employee evaluation system 
identifies the criteria that are being used to make decisions about employees. Many people do make 
employee decisions using biased, unreliable, and invalid information. Some people do this with the 
best of intentions. These people are conscientious supervisors who think that subjective information 
is the only information available. Others may understand the limitations of the information they are 
using but have no intention of pursuing objective information because this information may cause a 
change in their decision making style. For example, a supervisor who dislikes a supervisee may not 
want to get objective information that shows that this employee is doing a good job in some areas. This 
information might take away some of the control of this supervisor. Consequently, formal performance 
evaluation systems can help the supervisors who are genuinely interested in the best possible decision 
but performance evaluation information will be of little value to someone who wants to make a decision 
based on biased information. Correspondingly, formal program evaluation systems can help to improve 
program outcomes for those who have a genuine interest in improving program outcomes 
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How do you begin a program evaluation effort? 

One of the most common questions in program evaluation efforts is where to start. The answer to this 
question is really simple. You must start with the mission and the goals of the organization you are 
evaluating. The reason for this approach is equally simplistic. If program evaluation is to determine 
whether or not the program is successful, you must know what success means for the program. In 
other words, if you are going to make a determination regarding whether or not the program is making 
progress, you must know what the intended destination is. This point is best illustrated in an old joke: An 
airline pilot on a large commercial flight comes on the intercom and says: 

"I have good news and bad news. The good news is that we are making great time on this flight; 
the bad news is that we are lost." 

Program evaluators should avoid making similar pronouncements about the status of evaluation efforts 
that are not based on clearly identified goals. 

However, program evaluations can and do occur at many 
different stages in the development of an agency. This situation 
may complicate the answer to the question of where to start. If 
a program evaluation effort is initiated at the early stages of the 
formation of an organization, it is probably reasonable to assume 
that the organization does not have a well-defined mission or 
purpose. Also, many organizations that have been in existence 
for years do not have a mission or purpose statement. 

What do you do if there are no mission and goals 
identified for the organization? 

If an organization does not have a mission and goals, the 
evaluation needs to help develop these mission and goal 
statements. A comprehensive evaluation must take into account 
the direction of the organization in order to determine if 
progress has been made! 

Why is there so much confusion about mission, goal 
and objective statements? 

There are many different rules about what vision, mission, goal, 
and objective statements are. Some say that mission and goal 
statements are the same things. Others say that goal and objective statements are interchangeable. 
Some organizations have good value and mission statements but have no goal or objective statements. 

In order to avoid confusion, these are the definitions we will be using in this book: 

1. A vision statement is general, lofty and inspiring. 

2. A mission statement is a brief (no more than one paragraph) identification of the organization's 
realistic activities. Some people prefer the term "purpose statement". The mission statement 


When Tom began a program 
evaluation effort at a Community 
Mental Health Center (CMHC) in 
Denver in 1974, the CMHC was like 
most centers. The director had a 
clear image of what the mission 
and goals were. The managers and 
staff and board managers also had 
an image of the center mission and 
goals. Unfortunately, these images 
didn't match. This was a formula 
for disaster. The director, staff, and 
board members were working toward 
different goals. This is one of the first 
challenges for a program evaluation 
effort - development of a mission 
and goals with broad organizational 
support. 
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is quite broad in scope so that it covers all possible "purposeful activity" of the organization 
(Ludden & Mandell, 1993). 

3. A goal statement defines operationally how the mission of the organization is being 
implemented. The goal statement should, by necessity, be more specific than the mission 
statement. The goal statement provides a focus and direction for the organization. It highlights 
those issues that the organization values. The goal statement should clearly identify what the 
organization wants to happen. For example, the goal of providing the most services possible 
within the allocated budget identifies a clear organizational desire. 

4. An objective statement is a much more specific determination of whether or not progress has 
been made in achieving the goal. Generally, the objective statement must meet the following 
criteria: 

a. Must be time-limited. 

b. Must specify a clear criterion on which to determine whether or not the objective has been 
met (e.g. 100 people must be admitted for mental health services). 

c. Must be specific to one and only one aspect of the goal. 

In sum, an objective statement should be SMART - Specific, Measurable, Actionable, Reasonable, and 
Time-limited. 

Some international organizations use different definitions. For example, this is a statement on the World 
Health Organization (WHO) Website: 

The objective of WHO is the attainment by all people of the highest possible level of health in the 
sense that "Health is a state of complete physical, mental and social well-being and not merely 
the absence of disease or infirmity" as enshrined in the WHO Constitution as one of the basic 
principles. WHO provides technical support to address the country's priority health issues within 
the purview of WHO core functions which relate to engaging and partnerships, shaping the 
research agenda, setting norms and standards, articulating policy options, catalyzing change and 
assessing health needs. We provide support mostly in policy planning and program development; 
human resources development; prevention and control of major communicable diseases, polio 
eradication, leprosy elimination; health promotion; healthy environment; and health technology 
and pharmaceuticals. 

The first part of this statement fits our definition of a vision statement. The second part fits our definition 
of a mission statement. However, we need a consistent and uniform set of definitions in order to begin 
to define an organization. So we will use the above definitions in this book. 

How does program evaluation interface with the vision, mission, goals, and objectives 
of a program. 

One of the most important aspects of an effective program evaluation system is the assurance that the 
program evaluation information will be utilized by an organization to improve services. Many program 
evaluations have failed to address this critical issue; the result of this serious error is that many good 
evaluation efforts have provided little practical information. More than one program evaluation report 
has ended up gathering dust in an administrator's office because this issue was not addressed. 
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One of the best ways to assure that this won't happen is to 
build the program evaluation around the goals and objectives 
of an organization. At a minimum, a program evaluation effort 
should determine progress toward the identified mission, goals, 
and objectives. In order to determine if a particular program 
has been successful, it is necessary to know what would be 
considered successful. As simple as these sounds, few programs 
ever take the time to identify what success is. Consequently, 
program evaluation results lead to few consequences and even 
more importantly, service programs languish with little sense of 
direction or clear identification of the intended outcomes. 


Once the goals of the program have been established, the 
objectives or intended outcomes can be determined and the program evaluation effort can be designed 
around determining whether or not the intended outcomes of the programs have been met. 

A typical vision statement might look like the following: 

Vision: The Vision of is a community that values and respects people and is 

responsive to their individuals needs, wants and desires for the enrichment of their lives. 

(West Virginia, 2001) 

A typical mission statement and goals for a mental health organization might look something like this: 

Mission: The Mission of is to provide necessary public funded mental health services 

for residents of southeast Denver so that mental health functioning of the residents is maximized. 

Goal 1: Provide services to those individuals that are most in need of mental health services. 

Goal 2: Provide as much mental health service as possible within available resources. 

Goal 3: Provide the highest quality mental health services. 

Goal 4: Provide a work environment that is conducive to high staff morale. 

(Barrett and Dehaan, 1980) 

Goals should be comprehensive enough so that they address the major values of an organization. 
However, the identification of more than six (6) goals for an organization may be counterproductive. 

Objectives should be more specific than goals and 
should detail how the organization will evaluate 
whether or not progress is being made. Objectives 
will be the measuring device to determine progress 
toward the stated goals. 


Vision 

Mission 

Goal 

Objective 


More general 


More specific 


Before Tom began work at the CMHC 
in Denver, the center had been 
producing a report on how they spent 
their time. A huge thick report was 
generated every month that showed 
how each staff member spent 
every 15 minutes. Unfortunately, 
this information was being ignored 
because the information was 
irrelevant to the center goals. 
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A typical set of objectives for this hypothetical organization might look something like this: 

Goal 1: Provide services to those individuals that are most in need of mental health services 


Objective a. 

Ensure that at least 90% of admissions meet the state of Colorado criteria for the 
most in need population for fiscal year 'X5. 

Objective b. 

Ensure that the percentage of admissions from each age groups matches the 
percentage of need for that age level (within 5%) for all age groups as determined by 
the state of Colorado population in need information for fiscal 'X5. 

Objective c. 

Ensure that the percentage of admissions in each geographic region matches the 
percentage of need for that region (within 5%) in each region as determined by the 
state of Colorado population in need information for fiscal 'X5. 


Goal 2: Provide as much mental health service as possible within available resources. 

Objective a: Ensure that at least 5,000 people receive mental health services for fiscal 'X5. 

Objective b: Ensure that at least 2,000 families receive mental health services for fiscal 'X5. 
Objective c: Ensure that there are sufficient units of service delivered to produce 5 million dollars 

in revenue for fiscal 'X5. 

Goal 3: Provide the highest quality mental health services. 


Objective a: 

Ensure that consumer reports of service outcomes average at least 5.0 (on a six point 
scale as determined by the MHSIP report card survey) for surveys collected in fiscal 
'X5. (MHSIP, 1996) 

Objective b: 

Ensure that consumers improve functional levels by an average of 5 points on 
the Colorado Client Assessment Record (CCAR) as measured at admission and at 
discharge from treatment for consumers discharged in fiscal 'X5. (CCAR, 1995) 

Objective c: 

Ensure that consumers attain their treatment goals by at least 75% as measured by 
the Goal Attainment Scaling system evaluated within three weeks of discharge for 
consumers discharged in fiscal 'X5. (Kiresuk, T.S. & Sherman, R.E., 1968) 


Goal 4: Provide a work environment that is conducive to high staff morale 


Objective a: 

Ensure that employee morale, as measured by the annual self-report survey, 
averages at least 5.0 on a six point scale for fiscal 'X5. 

Objective b: 

Ensure that employee turnover is no more that the industry standards for the region 
as reported by Mountain States Employers Council during fiscal 'X5. (Mountain States 
Employers Council, Inc., 2003). 


Objectives should be reviewed at least annually and modified as necessary. 

Implementing objectives often requires a timetable, specification of available resources, and 
identification of personnel who will achieve the objective(s). Many times a key individual will be 
designated to head the implementation. 
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How do you maximize the impact of program evaluation? 

Many program evaluation efforts fail to reach their full impact because the results are not utilized to 
improve services. In addition to designing the evaluation system around goals and objectives there are 
certain principles that should be followed in order to maximize the impact of any program evaluation 
effort. 

1. Identify the people who are in the position to make decisions and solicit their support before 
program evaluation begins. Program evaluation efforts often fail to recognize the importance 
of determining who will be utilizing the program evaluation results. This failure will likely lead 
to either chaos or stagnation when the results are released. A better approach is to analyze the 
formal and informal decision-making system within an organization and identify key decision 
makers so that there is no confusion when the time comes to make decisions. 

2. Identify some potential scenarios for what the program evaluation might show and talk about 
what the consequences might be of any particular program evaluation result. This practice has 
the tendency to reduce anxiety as many people are suspicious of what the program evaluation is 
likely to show. The decision making process that should follow the release of program evaluation 
results can be difficult even when decision-makers have been identified beforehand. This 
decision-making process can be simplified by discussing scenarios and alternatives before the 
data collection begins. This discussion should include consideration of what information will be 
necessary to make important decisions about the program. For example, if a program evaluation 
study shows that the intended results for a particular program are not being achieved, what 
decisions will be made? Will more information be required before a program is modified or 
eliminated? If so, can this information be collected through the proposed data collection effort? 
Many potential problems can be avoided through these discussions. 

3. Focus the evaluation efforts on outcomes. This emphasis will avoid controversy about what are 
appropriate procedural standards and will ensure that participants understand that the ultimate 
goal is to increase outcomes. 

4. Report information to those that are in a position to affect the results. Many program evaluation 
efforts have failed because the reporting system goes only to administrators. Effective evaluations 
ensure that information goes to all levels within an organization. For example, as an evaluator 
within a CMFIC, Tom spent a considerable amount of time establishing a mechanism for ensuring 
that information on performance went to clinicians. Staff was more receptive to this practice 
when information was individualized so that performance information was broken down by 
individual clinician. Although this effort took a lot of time, the benefit of having information go 

to clinicians is critical. It is unlikely that organizational performance will improve from program 
evaluation data unless the information gets to the people that are in the best position to improve 
the performance. 

5. Ensure that evaluation information is audience-specific. Reports to the board of trustees should 
be less detailed than reports to the managers of a specific program. Ensure that the information 
reported is the information desired and needed by that level of the organization. 

6. Ensure that the program evaluation system is flexible enough to handle anticipated changes in 
goals and objectives for the organization. 
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Needs assessments (sometimes called environmental scans) are one of the most critical aspects of 
program evaluation. If mental health services are to be effective, they must be geared to address 
the needs of the intended target audience. The best services will not achieve their intended impact 
if the services don't address these needs. Consequently, objective needs assessments are critical for 
determining these needs so that appropriate services can be provided. 

What is the working definition of needs assessment? 

Needs Assessment is a systematic determination of level and/or type of need. While a variety of 
activities can be included under the title of need assessment, this discussion is focused on objective and 
systematic needs assessment efforts. 

When was the first need assessment? 

In order to answer this question we need to go back to the first program evaluation, which is clearly 
documented in the bible. 

“In the beginning when God created the heavens and the earth... And so it happened: the earth 
brought forth every kind of plant that bears seed and every kind of fruit tree on earth that bears 
fruit with its seed in it. God saw how good it was." ( Genesis 1:1,12) 

God did the first program evaluation by determining that the creation was good. He may have used 
different instruments than those we use today, but he did a program evaluation. 

"Lord God formed out of the ground various wild animals and various birds of the air, and he 
brought them to the man to see what he would call them; whatever the man called each of them 
would be its name. The man gave names to all the cattle, all the birds of the air, and all the wild 
animals, but none proved to be a suitable partner for the man." (Genesis 2:18) 

God did at least two need assessments and two corrective actions. He first determined that man had 
needs that were not met with the things God had created up to that point. So, God created animals and 
birds and did another need assessment and determined that man still had needs that weren't met. Then 
God went on to create something else to address the need for a suitable partner. This was the beginning 
of a continuous need assessment on the "suitability" of this partnership! 

So needs assessment has been happening for a very long time! There have been different labels applied 
to the practice of needs assessment, but it is such a critical activity that it will always be part of a 
comprehensive program evaluation. 


When should a major needs assessment be conducted? 

There is no "bad" time to do a need assessment, but some times are better than others. Needs 
assessment can be an expensive and time-consuming affair, so it is important to ensure that the 
information that is generated will be utilized appropriately. The following situations are times when 
needs assessment information is at a premium: 
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1. Before service delivery begins. 

2. When services are being expanded or curtailed. 

3. When the organization is reorganizing. 

These are good times to do needs assessment because the organization should be eager to utilize the 
information obtained from the needs assessment. The organization should be prepared to modify its 
programs at these times. The process of collecting and analyzing needs information can be expensive 
and time consuming. It is important that the information be collected at a time when there is the 
greatest likelihood that the information will be utilized. 

What are the different types of needs assessment? 

All needs assessments can be classified into six different types of methodologies (NIMH, 1976): 

1. Key Informant - A key informant survey is exactly what the title indicates - utilizing key 
individuals to inform about community needs. Individuals can be chosen for this survey by 
a variety of different methodologies. However, whatever selection process is utilized, some 
important principles should be followed. 

a. The usefulness of these surveys is entirely dependent on the respondent selection process. 
While this methodology does not assume a random selection process, it does assume that 
the respondents come from different perspectives. Consequently, the goals should be to 
choose individuals from a variety of different positions and settings. 

b. Correspondingly, individuals should not be chosen solely on the basis of their knowledge of 
services provided, since one of the goals of this process is to gain new information. 

2. Community Forum - The community forum methodology is based on the democratic principle 
that everyone should have an opportunity to impact the programs offered. Consequently, 
participation in this process is open to everyone. The basic format to this approach is similar to 
a town meeting where all those attending have an opportunity to express their opinions on the 
needs of the community and how well the agency is addressing those needs. 

3. Focus Group (sometimes called nominal group) -The focus group approach is similar to a 
community forum in that it involves a public meeting. However, the focus group approach adds 
one important component - structure. Focus group attendees are invited to the meeting. Also, 
there are structured questions for the group to answer. And there is a set process for getting 
individual and group feedback on the identified questions. 

4. Rates of Treatment - This particular need assessment approach involves the collection of 
information about who is utilizing current services. Although this approach provides little 
information on unmet need, it is a useful approach when combined with other methodologies 
that are intended to assess the total need. For example, an epidemiological survey might 
produce information on the total population with mental health needs. The rates of treatment 
information can then be subtracted from this total to get an estimate of unmet need. 

5. Social Indicators/Social Determinants - The social indicator approach utilizes existing research to 
identify variables that predict mental health need. For example, studies suggest that areas that 
have elevated school dropout rates have high mental health needs. This finding does not suggest 
individuals that drop out of school have mental health needs. These studies merely confirm 

a correlational relationship. An example of this is Healthy People 2020. (<healthypeople.gov> 
website accessed July 7, 2014.) 
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6. Survey or Epidemiological Survey - The survey methodology is considered to be the most 
sophisticated form of need assessment because it utilizes a direct survey of a sample of the 
population. The survey can take the form of a mail-out, a face-to-face survey, a phone survey, or 
an internet survey. 

What follows is a chart of these six different methodologies. Under each title is a definition, followed by 
the steps utilized in implementing that particular methodology, followed by some of the advantages of 
using this methodology, followed by some of the disadvantages. 


Type 

Steps 

Advantages 

Disadvantages 

1. Key Informant 

Collection of input from select 

a. Prepare instrument 

Public relations 

Sampling problems 

community leaders and 
representatives 

b. Select respondent 

c. Collect data 

d. Analyze 

e. Provide feedback to respondents 

Low cost 


2. Community Forum 

Open Town Meeting 

a. Advertise 

Cost-effective 

Potential for 


b. Select leader 

c. Convene the forum 

d. Collect data 

e. Analyze 

Can build support 

negative public 
relations 

3. Focus Group (Nominal Group) 

Structured Workshop to answer 
predetermined questions 

a. Prepare format 

b. Select group 

c. Convene the group 

d. Collect information 

e. Analyze and provide feedback 

More focused 

Miss some information 


A typical sequence of events for a nominal group meeting might look like this 

a. Introduction 

b. Problem description 

c. Group round robin 

d. Group discussion 

e. Selection and ranking 

f. Group tally 

(May repeat for large group) 

4. Rates Under Treatment 

Analysis of current utilization 

a. Identify agencies 

Less subjective 

Only current users 

of services 

b. Collect data 

c. Analyze 

d. Provide feedback to participating 
agencies 

Current users are 
evaluated 
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Type 

Steps 

Advantages 

Disadvantages 

5. Social Indicators 




Need determination based on 

a. Selection of indicators 

More objective 

Comparability? 

descriptive statistics and 

b. Collect data 

Simple 


assumptions about relationship 
between statistics and need 

c. Estimate need 



6. Survey 




Direct assessment of need from 

a. Select instrument 

Sampling can 

Still based on the 

interviews of a sample of the 

b. Obtain and train interviewers 

extrapolate to 

inviewee reporting 

total population. An example 

c. Select sample 

population 


of this is the National Comorbidity 

d. Collect data 



Study (NCS). (Kessler et al, 2005). 

e. Complete verifications 

f. Analyze data 




How do you choose from amongst all the different methodologies? 

There are several important factors to consider in the selection of need assessment methodologies. 
Probably the most significant factor is to insure that there is a good fit between the selected 
methodology and the information that you hope to obtain. You do not want to conclude at the end of 
your study that you did not obtain the correct information because you utilized the wrong methodology. 

Another important consideration in the selection of methodologies is to ensure that you are 
utilizing methods that will complement each other. It is almost always necessary to utilize multiple 
methodologies in order to get a comprehensive picture of what the needs are. The selection of 
methods should be done in such a manner as to ensure that the different methods don't duplicate but 
complement each other. For example, it would be foolish in most circumstances to conduct both an 
open town meeting and a community forum. These two methodologies are very similar and have much 
the same advantages and disadvantages. Conversely, key informant and social indicator approaches have 
very different advantages and disadvantages and will likely complement each other so that each method 
will bring in new information. 

Needs Assessment in International Settings 

Needs assessment is universal; it is just as critical in international settings as in the United States. 

The most common needs assessments in low and middle income countries are key informant, 
community forum, nominal group and community survey, because there is rarely enough information 
to do the rates under treatment or social indicators approaches. The key informant methodology is 
regularly used and the community survey methodology is utilized where there are sufficient resources. 
Key informant methodology is important in international settings because community leaders are often 
in the best position to identify what the needs are. Also, it is critical to ensure that community leaders 
are supportive of the plan to deliver services. 


18 


Human Services Program Evaluation 

Needs Assessments 


Chapter 3 


In 2012 the World Health Organization and the UN Refugee Organization (UNHCR) released a document 
titled: Assessing mental health and psychosocial needs and resources; toolkit for humanitarian settings 
(WHO, 2012). This document is a good resource for identifying some useful tools for completing needs 
assessments in countries around the world and can be downloaded from the WHO website. 
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Appendix 1 

Guidelines and criteria for constructing surveys 


Much of needs assessment is accomplished through the use of surveys. Often there are existing surveys 
(with established validity and reliability) that can be used for a particular project. However, more 
frequently, the needs assessment requires the development of one or more custom surveys. These 
guidelines are provided for use in these circumstances. 

• Is a questionnaire the best way to obtain your necessary information? 

• What are the qualities of a good questionnaire? 

• How do you go about designing a standardized questionnaire? 

• What are the modes of administration? 

• What is a systems approach to the design of a questionnaire? 

• How do you compose questions? 

Most of us will be asked during our careers to gather data about clients, customers, families, employees, 
suppliers, creditors, regulators or some other stakeholder group. If JD Powers did not have a ready 
response you can use, you may be compelled to gather the data yourself. How do you do it? Read on. 

Questionnaires as a Life Line 

On the other hand, you may be asked to participate in a survey. Should you commit your time and your 
organization (often time again) to respond to a questionnaire? How do you evaluate the research effort? 
Does the survey instrument reflect sound design? Read on. 

Gathering information from key stakeholders can be gleaned from direct contact, comment cards, 
formal surveys, focus groups, field intelligence, complaints or the Internet. The design of any of these 
information sources spells out its potential usefulness. There are at least 10 questions you should answer 
in deciding to use a questionnaire in your research or in deciding to respond to a questionnaire. The 10 
key words are the maps to our analysis. 

► About Knowledge 

Do the respondents have the information (or knowledge) about the research issue on hand? 

► About Interest 

Are the respondents willing or ready to reply? Or in a word, do they have the interest to participate? 

► About Attitude 

What is the direction of opinion toward the issue(s)? What is the direction of their attitudes? 

► About Feelings 

What are the reasons for their feelings? Why do they feel the way they do? 

► About Strength of Feelings 

What is the intensity of the attitude? Or, how strongly do they feel about the issue? 
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► About Saliency 

Is the issue on the respondent's mind? Is the issue salient or relevant to the respondent? 

► About Expectations 

How does the respondent's view of the future affect present attitudes? Or, what are his/her 
expectations? 

► About Readiness to Act 

What are the differences between the respondent's attitudes and behavior? Is he/she ready to act 
on his/her attitudes? 

► About Perceptions of Other's Beliefs 

What is the perception of other's belief(s)? What is the level of pluralistic ignorance? 

► About Other Options of Inquiry? 

Are other options of inquiry such as direct observation, a field experiment, or content analysis likely 
to produce better information? 

Qualities of a good questionnaire 

A good questionnaire exhibits three major qualities. First the questionnaire meets the objectives of 
the research. Second, it obtains the most complete and accurate information possible. Third, the 
questionnaire abides within the time and resources limits of the project. 

Designing a standardized questionnaire 

In some instances, standardized questions may inappropriately narrow the discussion. In these cases, the 
major questions should be used with appropriate probes to obtain the desired information. In this case, 
the major questions are used an interview guide. 

Standardized questionnaires have the following advantages. They 

• measure and control response effects. 

• reduce unacceptable or un-codable responses. 

• help the researcher avoid being overwhelmed by a mass of idiosyncratic material. 

On the other hand, the disadvantages of a standardized questionnaire focus on the respondent. The 
respondent 

• may understand the question differently from your intent 

• may be forced into perceived unnatural responses 

• has no opportunity to qualify his/her answers or to explain more a response more precisely 

• may feel s/he has already answered the question when follow-up questions are asked. 
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Mode of administration 

The administration of the instrument can follow one of two major paths: 

1. Self-administration with a 

• mail survey that offers 

- low cost, no interviewer time or costs, and efficient data gathering if the responses are 
from specialized and/or highly motivated groups, however, 

- response rates are usually low, biases can be large, and someone other than the 
identified subject may complete the instrument. Questions about the instrument cannot 
be answered. 

or 

• group of respondents with the interviewer present so 

- questions about the instrument can be answered with a resulting high completion rate, 
but there may be 

- higher costs and potential bias if using convenience or volunteer samples. Generally this 
is an inefficient approach to obtain generalizability. 

2. Personal interviews: face-to-face or telephone interviews 

• Questions can be answered and completion rates are usually high. A full exploration of 
the respondent's views is possible and unexpected answers or unforeseen factors can be 
followed-up. The sessions can be taped and analyzed qualitatively (rather than statistically). 

• Interviews need to limited in length and complexity because of time and costs factors and 
usually requires a trained individual. 

A systems approach to the design 

Using a systems approach to the design and deployment of a research questionnaire, five major issues 
emerge: 

1. What are the major questions to be answered? (Queries about the information needed) 

2. What is the content and layout of the questionnaire? (Questions that respond to the queries) 

3. How will the output look? (Table layout of rows and columns, composition of graphs, etc.) 

4. How does the instrument link to the output? (Linkages of questions to table or graphics) 

5. What are the limitations on the findings and interpretations from the study? (Pitfalls or 
shortcomings in the analysis). 


Preparing the questionnaire (with only 17 steps!) 

Issue 2 (along with a lot of hard work) requires the formulation and layout of the questions and the 
questionnaire. Several key steps involve 1 


1 Adapted from Seymour Sudman and Norman M. Bradburn, Asking Questions, Jossey-Bass, Publishers. 1982. pp. 281-282. 
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1. Searching data archives for existing questions and scales on the topics of interest. 

2. Drafting new questions and/or revise existing questions. 

3. Preparing the introduction. 

4. Preparing the instructions. 

5. Formatting the questions (including logical groupings). 

6. Applying the ADEQUACY CRITERIA CHECKLIST and making appropriate corrections. Please see 
Figure 1 at the end of this technical note. 

7. Performing precolumn and precode of the questions (namely, questionnaire data capture layout). 

8. Obtaining peer evaluation of draft questionnaire (either in a group or individually). 

9. Revising the draft and testing the questionnaire on yourself and colleagues. 

10. Preparing sample interviewer instructions for pilot test. 

11. Pilot-testing on a small sample of respondents (20 to 50) similar to the universe from which you 
are sampling. 

12. Obtaining comments of interviewers and respondents in writing or in debriefings. 

13. Eliminating questions that do not discriminate between respondents. 

14. Revising questions that cause difficulty. 

15. Applying the ADEQUACY CRITERIA CHECKLIST and making appropriate corrections. See Figure 1 
again. 

16. Pilot-testing again. 

17. Preparing final instructions. 

Composing the Questions 

Questions may be open or closed. Open questions allow respondents to answer in their own frames of 
reference and to reveal what is most salient to the respondents, but may elicit repetitious and irrelevant 
information or hard to combine responses and, in an interview format, may require special training of 
the interviewers. 

Open questions work well: 

• When there are too many categories to be listed or foreseen. 

• When a spontaneous and uninfluenced reply is wanted. 

• When there is a need to build rapport. 

• When doing exploratory interviewing and pretesting to eventually close up the questions later on. 
Closed questions: 

• Enhance understandability. 

• Ease coding and analysis. 

• May provide answers the respondent did not think of. 

• May force respondents into an unnatural response. 

• May cause respondents to choose responses not reflecting the exact shade of their opinion. 

Response categories should vary depending on the objective of the question. The following table 
illustrates typical responses and related application. 
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Type of Response 

Application 

Excellent-good-fair-poor 

Ratings 

Approve-disapprove; favor-oppose 
(Expand with how strongly do you feel) 

Opinions 

Agree-disagree 

(Expand with how strongly do you feel) 

Statements or Propositions 

Too many-not enough-about right 

Satisfaction, attention, amounts 

Better-worse-about the same 

(Expand with very much better-sl ightly better, etc) 

Comparisons with past or future 

Very-fairly-not at all 

Importance, interest, satisfaction 

Regularly-often-seldom-never 

Frequency 

Always-most of the time-some of the time-rarely or never 

Frequency 

More likely-less likely-no difference 

Probability of action 


Question writing guidelines include: 

• Keeping it simple. 

• Avoiding lengthy questions (25 words or less). 

• Specifying an alternative (e.g., "Does this seem like a good idea to your" [Yes or No] versus "Does 
this seem like a good idea or a bad idea?". 

• Loading questions only to enhance truthful responses (e.g., How old were you the first time you 
masturbated?"). 

Common errors in composing questions include the following: 

• Double-barreled questions: "Are the media doing a responsible and fair job of covering 
Amendment #11 on Workers Compensation?" [responsible: accountable, competent, reliable? 
fair: impartial, average, mediocre?]. 

• False premise: "To make more basic research design textbooks available, publishers should raise 
the prices on current text offerings?" [Respondent doesn't want any more basic research design 
texts!]. 

• Vague words: "Do you read the Journal of Behavioral Health Services & Research ?" vs "Do you 
read the Journal of Behavioral Health Services & Research regularly, occasionally or never?" vs 
"How many times have you read the Journal of Behavioral Health Services & Research in the last 
year?". 

• Overlapping alternatives: "Are you generally satisfied with your career in educational counseling 
or are there some other things you don't like?" [Respondent cannot clearly answer yes or no.]. 

• Double negatives: "Would you favor or oppose the AMA passing a resolution not allowing 
managed care firms to force employees to attend AMA sponsored Continuing Professional 
Education (CPE) courses?" [What are you for or against?]. 

• Intentions to act: "Do you intend to stay with your current employer?" vs "In five years, do you 
expect to be with this employer?" [Increase precision]. 
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Introduction, question order and format 

The introduction, question order and format are important considerations. The introduction is crucial 
and may be on the questionnaire or in a one page introductory letter or telephone call or FAX or e-mail. 
The introduction should cover: 

• What the study is about and its social usefulness. 

• Why the respondent is important. 

• Promise of confidentiality. 

• Obtaining informed consent. 

• Reward for participation. 

• What to do if questions arise. 

• Thank you! 

The order of the questions should consider the following suggestions: 

• Initial questions are easy and non-threatening. 

• Threatening question are later ["Did you kill your brother?" and demographics about age, 
income, etc. are at the end], 

• Group like questions together. 

• Use a shift in attention to move to new topics ["Here are more questions regarding ..."]. 

• Generally start with a broad question about a topic and then ask more specific questions: 
exception: complex issues or ones the respondent has not thought about. 

Good format suggests the individual items are spaced appropriately with related items grouped together 
(already mentioned!) and sufficient space exists for response categories. 

Will following these guidelines and criteria guarantee success? Probably not, but you will have less 
rework! 
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Figure 1. 

Adequacy Criteria Checklist 2 


A. Title 

] Title reflects content of the instrument. 

] Title is concise. 

] Title is written in language easily understood by the participants. 

B. Introduction 

□ There is a clear statement of the instrument's purpose. 

] Respondent is told why s/he was selected to participate. 

] Respondent is told how information will be used. 

] Privacy of confidential information is assured. 

] Anonymity of respondent is guaranteed (if appropriate). 

] Informed consent is acquired. 

] Motivators for responding are provided. 

] Directions include an estimate of the time required to complete the instrument. 

] A return date is specified for the return of the instrument. 

C. Directions 

] Directions are given for each section (if necessary). 

□ Language used is appropriate for the level of the respondents. 

□ An example item is provided (if necessary). 

] Directions are given for completing accompanying answer sheet(s). 

] Directions are provided for responding to items which "do not apply". 

D. Format 

] Individual items are spaced appropriately. 

] Related items are grouped together. 

] Sufficient space exists for response categories. 

] Demographic items are listed last (usually). 

] Instrument is reproduced clearly. 

E. Item Stems 

] Stem is relevant to the stated purpose of the instrument. 

□ Stem is unidimensional. 

□ Wording of the stem is appropriate for the reading level of the respondent. 

] Possible response is not biased by the wording of the stem. 

] Stems requiring respondent to supply information identify the appropriate unit response. 
] Each stem is independent of other stems. 

F. Responses 

] Response categories are relevant to the stem. 

] Response categories are unidimensional. 

] Response categories are non-overlapping. 

] Response categories are exhaustive. 

] Response sets are written in the same direction. 

] "No applicable" options are provided where appropriate. 

□ "I do not know" options are provided where appropriate. 

□ "No opinion" options are provided where appropriate. 

] Sufficient space is left for responses provided by the respondents. 

] Guidelines for comments are presented (when appropriate). 


2 Adapted from Robert W. Covert, Guidelines and Criteria for Constructing Questionnaires, University of Virginia, Charlottesville, VA, 1977, 
and Dan Manzares and James E. Sorensen, Monitoring the University, University of Denver, Denver, CO, 1989. 
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Client Outcomes and Costs in Human Service Organizations 1 

Overview 

Managers of human service organizations are expected to acquire and use resources to create effective 
and efficient services. Efficient cost management requires understanding cost behavior, applying cost 
distinctions for planning and control, computing unit-of-service costs, using unit-cost data in contracting 
and financial management and adding credibility to the unit-of-service costs by including the opinion 
of an independent auditor. But the fast emerging managed care environment requires more than just 
efficient cost management. Managed behavioral health care seeks to reduce or eliminate unnecessary 
services, reduce and control the costs of care, and maintain or increase outcomes and effectiveness. As 
costs are reduced, concerns surface about compromised quality of care or, more specifically, poor clinical 
outcomes and meager client satisfaction. Knowing about clients' contentment with services can help 
identify costs to be enhanced, diminished or reengineered. Outcome measures such as client functioning 
or symptomatic psychological distress or quality of life appropriate for the age and client type should 
be considered. Client satisfaction should also be measured. While not a measure of client functioning, 
assessing client satisfaction is a key measure of program performance and may be as important as 
treatment outcome. Standardized methods provide the ideal assessment approach. Comparing the costs 
and outcomes of two or more services enables managers (and policy makers) to make cost-effective 
choices among services and programs. Human service programs must document costs, outcomes and 
client satisfaction at a minimum to survive the assault of managed care. As part two of a three-part 
series on human service that includes (1) analyzing cost dynamics, (2) linking costs and client outcomes, 
and (3) choosing cost-effective management strategies, this report builds a framework outlining the role 
of costs and outcomes in cost-outcome and cost-effectiveness analyses. This chapter explores issues 
related to outcomes (to be linked with costs) and how cost-outcomes and cost-effectiveness may be 
used as a management strategy in operating human service programs. 

Human service programs face many challenges. Funding methods are shifting as Medicaid expenditures 
grow and federal funds to states and local governments are concurrently cut. Rural areas suffer from 
a lack of health-care resources (existing resources are often underfunded and understaffed) and an 
absence of integrated health-care systems. New service systems are developing with a shift from 
provider-centered to client-centered services. Pressures are increasing for assessment of client and 
program outcomes and effectiveness. Demands are also increasing for more organized and efficient 
services, all resulting in a thrust toward managed care (Broskowski 1991; Feldman 1992; Woodward, 
1992; Wagenfeld et al. 1994; Minden 1996; Manderscheid and Henderson 1997; Heflinger and Northrup 
2000; Fuchs 2000; Dorfman and Smith 2002; Hudson and Chafets 2010). Service settings grow more 
complex when a client has a dual diagnosis - mental health and substance abuse or mental health and 
developmental disabilities (Alterman et al. 1993; Solomon et al. 1993; National Association of State 
Directors of Developmental Disabilities Services (NASDDDS) 2003). Medicaid is likely to play a key role in 
managed care, especially for those with severe mental illness (Glazer and Goldman 2009, and Shern et 
al. 2008). 

Coping with these constraints and opportunities in a human service setting requires both efficiency and 
effectiveness. Efficiency is the accomplishment of objectives at a minimum cost, while effectiveness 

1 This chapter is supported by CMHS/SAMHSA Contract No. 280-94-0014, Ronald Manderscheid, Ph.D., Government Project Officer. This 
chapter is available on the Western Interstate Commission for Higher Education website: www.wiche.edu/MentalHealth/Frontier/ and 
under Frontier Mental Health. It also appears in the Journal of the Washington Academy of Sciences (2000). Vol. 86 #3, 159-177. 
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measures how well objectives are achieved. This paper focuses on how the key tools of cost analysis, 
clinical outcomes, client satisfaction, cost-outcomes, and integrated health and human services can aid 
the manager of human service programs in the developing managed care environment. 

Managed care environment 

Managed care is now an omnipresent pressure in health care (American Managed Care and Review 
Association or AMCRA, 1995). While some states with large frontier populations show low penetration 
of managed care, states with somewhat smaller frontier populations show higher utilization of managed 
care. To illustrate, North Dakota and Montana (both with more than 25% of their population in frontier 
counties) reveal 0.8% and 1.6% of the state population covered by managed care while Colorado (with 
5% of its population in frontier counties) and California (with less than 1% of its population in frontier 
counties) have over 76% and 58%, respectively, of the state population in managed care (AMCRA, 1995). 
State mental health authorities (SMHAs), for example, are reporting widespread current (or intended) 
use of managed care operations or contracts and the use of Medicaid funds to finance managed care 
(Sherman, Zahniser, and Smukler 1995; Chuang et al. 2011). 

In human services, managed care seeks to reduce or eliminate unnecessary services, reduce the costs 
of care and maintain or increase effectiveness. The thrust is to improve client outcomes, control costs, 
and decrease system fragmentation. Managing care limits services to necessary and appropriate 
care delivered by qualified providers in the least restrictive setting at considerable cost savings when 
compared with unmanaged care (Freeman and Trabin, 1994). The move toward managed care brings 
pressure to reorganize the service delivery system to improve client access, improve and augment 
service, and streamline administration. The goal is human services equal to or better than in the past for 
less cost and with more accountability. 

Managed care is not without its critics. Miller and Luft (1994) found, 
as an example, unfavorable mental health outcomes in HMOs 
they studied. The poor HMO results may be partially explained 
by the prepaid plan's comparatively early discontinuation of 
medication (Rogers et al. 1993). In another study, managed care 
gatekeepers failed to correctly identify depression in nearly 60% 
of the cases under review (Wells et al. 1989). A recurrent theme 
is outpatient treatment as a less expensive alternative to inpatient 
treatment (McGuire and Frisman 1983). While curbing expensive 
inpatient services, outpatient services, as a preferred alternative to 
inpatient care (Kiesler 1992), may be effectively discouraged if the 
outpatient services are limited and/or linked to high co-payments 
(Miller 1995). A key issue is the assurance of an appropriate level 
of outpatient treatment while still managing costs (Floward et al. 

1993. Schlesinger et al. 1983). If managed care makes a good-faith effort to curb abuses, rectify ethical 
problems, and address treatment effectiveness issues, it could offer superior performance to fee-for- 
service medicine (Boyle and Callahan 1995). Managed behavioral health care in its various forms appears 
to reduce costs and improve access, but the effect on quality has not been conclusively demonstrated 
(Minden and FHassol 1996). Reduction of costs in public sector managed behavioral health care programs 
also remains inconclusive (Minden and Flassol 1996). 


"Rather than the specific care model, 
structure or for-profit/nonprofit 
status, it is often contractual 
requirements, fiscal incentives, 
oversite, and leadership that have 
the most significant impact on how 
a managed care plan will meet the 
needs of children and adults living 
with mental illness and co-occurring 
substance use or primary care 
disorders (NAMI 2011)." 
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A comprehensive view of human services 

With the stimulus of widespread implementation 
of managed care, various healthcare organizations 
are focusing a comprehensive framework of 
analysis (see sidebar) using broad spheres of 
activity (or influence) called domains (MHSIP 
1996, 2013; ACMHA 1997, 1998, 1999; 

NASMHPDRI 1997, 1998). While the list varies 
across organizations, the domains include 
generally The MHSIP Consumer-Oriented Mental 
Health Report Card (1996) list of four: 

• Access - is a full range of needed services 
quickly and readily obtainable? 

• Appropriateness - do appropriate services 
address a consumer's individual strengths 
and weakness, cultural context, service 
preferences and recovery goals? 

• Outcomes - do services for individuals 
with emotional and behavioral disorders 
have an effect on their well-being, life circumstances, and capacity for self-management and 
recovery? 

• Prevention - do preventive activities reduce the incidence of mental disorders by (1) early 
identification of risk factors or precursor signs and symptoms of disorders and (2) increasing 
social supports and coping skills for those at risk? 

An analysis of each domain produces a robust set of categories and questions. Tablel illustrates a rich 
resolution. The MHSIP analysis of the domains focuses heavily on the customer perspective. What is the 
customers' perception of access, appropriateness and outcomes? Besides the customer viewpoint, a 
human service manager may want additional measures. For example, access includes continuity of care, 
integration of physical and behavioral health care, use of hospitalization, success at engaging specific 
target populations (or penetration rates) and assessments of waiting time. The other domains expand in 
the same way. A suggested expansion is shown in Table 1. Many of the questions surrounding a domain 
are pervasive and may emerge at a service or program level or, perhaps, at a county or state level. 

When a frontier service provider tries to select from the bewildering number of interesting and relevant 
questions, s/he is compelled to make choices because of limited resources, especially time and money. 
This paper suggests several questions may be more important than others given sparse resources and 
two are prominent: service costs and client outcomes. 

As human services increase as a part of total health services (Broskowski 1991), new emphasis is placed 
on costs and outcomes (Mirin and Namerow 1991). Managing care requires careful documentation of 
the costs of services and of clinical outcomes. Strategies to monitor and assess treatment plans and 
outcomes take many forms ranging from preadmission reviews, continuing treatment authorizations, 
concurrent review, screens (often computerized), to performance outcome measures (Austin, Blum 
and Murtaza 1995). This documentation of cost and outcome can be used, in addition, to respond to 
consumer and management concerns. Now consumers (including clients, employers and payers) are 


The National Association of State Directors of Developmental 
Disabilities Services (NASDDDS) and the Human Services Research 
Institute (HSRI) launched a Core Indicators Project in 1997. The 
name of the data collection collaborative was changed to National 
Core Indicators (NCI) in 2002. The aim of the initiative is to develop 
nationally recognized performance and outcome indicators that 
will enable developmental disabilities policy makers to benchmark 
the performances of their state against the performance of other 
states. National Core Indicators also enable each participating state 
developmental disabilities agency to track system performance and 
outcomes from year to year on a consistent basis. NCI also uses 
domains: 

Domain 

Subdomain 

Consumer Outcomes: 

Work, Community Inclusion, 
Choice and Decision-making, 
Supporting Families, Family 
Involvement, Relationships, 
Satisfaction 

System Performance: 

Service Coordination, Utilization 
and Expenditures, Access 

Health, Welfare & Rights: 

Safety, Health, Respect/Rights 

Service Delivery System 
Strength and Stability: 

Acceptability, Stability, Staff 
Qualifications/Competency 
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Table 1. Analysis of Domains 


Domain 


Illustrative focus of content 


Access 

1 . 

Consumer survey 

1 . 


2. 

Continuity of care 



3. 

Integration of physical and 

2. 



behavioral health care systems 



4. 

Hospital utilization 

3. 


5. 

Penetration rates 



6. 

Waiting time 

4. 


5. 


6 . 


Appropriateness 1. Consumer survey 1. 

2. Continuity of care 

3. Cost of services 2. 

4. Integration of physical and 

behavioral health care systems 3. 

5. Voluntary participation 4. 

6. Penetration rates 

7. Services to promote recovery 5. 


6 . 

7. 


Illustrative type of question 

What are the customer's perceptions of 
access? Complaints? 

What are the arrangements to refer inpatients 
to residential or outpatient services? 

How is the transition from one to the other 
supposed to work? 

What are readmission rates and average 
lengths of stay (ALOS)? 

What is the ratio of x clients served to total x 
population in catchments area? 

What are the standard and actual results for 
timeliness after request for services? 

What are the customer's perceptions of 
appropriateness? Complaints? 

What are the referral patterns from inpatients 
to residential or outpatient services? 

What is the cost per unit of service? 

What are the numbers of coordination events 
between the two? 

What is the percentage of inpatient admissions 
that is involuntary? 

What percentage of certified SPMI is served? 
What is the ratio of residential to inpatient 
units of service? 


Outcomes 


Prevention 


1 . 

Independence 

1 . 

What is the average number of days spent in 

2. 

Criminal justice 


the community? 

3. 

Productive activity (employment 

2. 

What is the proportion of adults and children 


or education) 


who spent time in jail? 

4. 

Functioning 

3. 

What is consumer's vocational and/or 

5. 

Hospital utilization 


educational status? (Days worked? $ earned?) 

6. 

Living situation 

4. 

What is the change in functioning over time? 

7. 

Quality of life 

5. 

What is the proportion of clients readmitted 

8. 

Satisfaction 


within 30 days? 

9. 

Substance abuse 

6. 

What is the type of living arrangement (and 

10. Symptom reduction 


level of independence)? 


7. What is the level of general life functioning? 

8. What is the consumer satisfaction with their 
mental health center and services? 

9. What is the age of first use of alcohol? 
Marijuana? Cocaine? 

10. What is the reduction in symptoms? 


1 . 


2 . 


Information provided to reduce 
the risk of developing mental 
disorders. 

Interventions designed to reduce 
the risk of developing mental 
disorder. 


1. What are the expenditures per enrollee on 
dissemination of preventive information? 

2. What is the percentage of enrollees 
participating in preventive programs? 
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beginning to demand accountability for the consumption of resources and the client outcomes in human 
service programs. Good managers of human service programs need to know how well their program 
and their clients are doing. Information systems (IS) to meet this need should focus on systematic cost 
reports, indicators to assess clinical outcomes, and analyses of costs and outcomes to evaluate cost- 
effectiveness. Comparing the costs and outcomes of optional services enables cost-effective choices 
among services and programs. Today's complex human service environment gives neither easy nor clear- 
cut guidelines for these information requirements. 

This chapter will focus on issues of measuring outcomes and links to costs. Regardless of the 
programmatic or service strategy taken, assessing the costs and outcomes is a vital first step in managing 
for cost-effective human service. 

Outcomes 

Concern for client outcomes was embedded in the traditional human service program evaluation 
literature (Attkisson et al. 1978; Ciarlo et al. 1986). Outcome has also become part of a larger quality 
movement in health care known as Continuous Quality Improvement (or CQI). In the corporate sector 
the movement is often called Total Quality Management (or TQM) and is associated with improvements 
in employee morale and productivity, customer satisfaction, and financial viability (GAO 1991; Ernst and 
Young 1992; ACHMA 2002). The CQI movement complements managed care as both focus on client 
outcomes. CQI in managed care calls for providing "... the right 
care ... delivered] to the right patients at the right time in the 
right way" (Freeman and Trabin 1994). A significant feature of this 
quality movement in health care is the reemergence of a concern 
for the client and how s/he feels about and responds to health care 
encounters. Shern (1994, p. 23) described the linkage between 
CQI and outcomes by observing "... CQI focuses on a recipient 
and outcomes orientation with an emphasis on understanding 
how program processes are related to desired outcomes." The 
application of CQI in human service, unlike health care, is in an 
early developmental stage (Rago and Reid 1991; Evans, Faulkner 
and Flodo 1992; Sluyter and Barnette 1995). 

How to conceptualize and evaluate successful implementation. 

Implementation outcomes distinct from service system and clinical 
treatment outcomes are still in process (Proctor et al. 2010). 

Taxonomies of implementation success (namely, implementation 
outcome, level of analysis, theoretical basis salience by stage of 
implementation and available measurements) "... pave the way to 
studies of comparative effectiveness of implementation strategies 
(Proctor 2010, 66). 

As purchasers and providers press prices and costs downward, 
consumer concern about compromised quality of care are likely to 
surface. Outcome management and practice guidelines programs 
may be able to deliver consistent and high quality care by reducing 
practice pattern variation (Freeman and Trabin 1994). 


Mental Health and Substance 
Abuse Benefits 

Mental health and substance abuse 
services should be included in health 
care reform because mental illness 
and substance abuse disorders affect 
more than 50 million Americans. Cost- 
effective treatments for mental illness 
and substance abuse disorder are 
available and high rates of success are 
being achieved across the spectrum of 
diagnoses. Private insurance coverage 
of mental health and substance abuse 
care discriminates between mental 
health and substance abuse care and 
general medical care by limiting the 
number of visits or days of treatment. 
Limits on access to care have resulted 
in shifting both the responsibility and 
cost of care onto the public mental 
health and substance abuse system 
[abstract from Aaron, B.S. (1993), 
SAMSHA News, 1 p. 11], 
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State-level Indicators. The National Association of State Mental Health Program Directors (NASMHPD) 
Research Institute compiled an inventory of managed care performance indicators including outcome 
measures for state mental health programs (Mazade 1997; NASMHPD Research Institute 1977). The 
data base should reflect service structures, levels of resources available, processes and outcomes used 
in developing and monitoring managed care contracts. In a five state feasibility study on state mental 
health agency performance measures, the NASMHPD Research Institute (1998) examined the feasibility 
and comparability of state performance indicators on: 

• outcomes (e.g., improvement of functioning, reduction in symptoms) 

• consumer evaluation of care (e.g., outcome, access, appropriateness) 

• consumer status (e.g., % employed, % living independently) 

• community services (e.g., % contacted within 7 days of hospital discharge, % receiving case 
management). 

In this study a human service organization could be responsive to state requirements for performance 
information if it did obtain outcome and consumer evaluation of care data and was able to extract 
consumer status (e.g., % employed) and community services information (% receiving case 
management) from internal sources such as the client record. 

Classifying outcome measures. Ciarlo et al. (1986) consolidated knowledge about outcome measures for 
human service clients. The authors suggest a useful three-dimensional taxonomy: 

• Assessment approach (individualized, partially standardized and standardized methodology) 

• Functional area/domain assessed (individual/self, family/interpersonal, and community 
functioning) 

• Respondent (client, collateral, therapist, and other) 

Client satisfaction with services is differentiated from client outcome evaluation because "... the former 
measures do not normally address any specified area of client functioning" (Ciarlo, et al, p.l). In the 
new thrust of managed care and CQI, however, the satisfaction of the client or an organization (e.g. 
Medicaid, an employer, or a managed care vendor) may be as important as treatment outcome (Ware et 
al. 1983). Competitive advantage accrues to providers who learn about and respond to customer needs. 
The challenge is to "... design an assessment program that provides useful, reliable, and valid data in an 
easy-to-use and cost-effective manner" (Plante, Couchman, and Diaz 1995, 265). Quality for rural areas 
may be meaningfully addressed through a combination of clinical outcomes and client satisfaction (Bird, 
Lambert, and Hartley 1995). 

Recommendations. Most human service programs should focus on outcome measures such as: 

• client functioning or symptomatic psychological distress or quality of life that are appropriate 
for the age (adult, adolescent, or child) and type (e.g., inpatient or outpatient, severely and 
persistently mentally ill, alcohol or other drug abuser) of patient, and 

• satisfaction of the client. 

Standardized methods provide the ideal assessment approach (Ciarlo et al. 1986). Well-standardized 
measures are needed to maximize the reliability (the extent to which the measure is reproducible) and 
sensitivity (the extent to which true changes in functional status can be detected). McLellan and Durell 
(1996) argue that standardized measures permit comparison conditions. Results from a single evaluation 
can be measured against results from a larger data base of comparable patients' samples and treatment 
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conditions. Without comparisons, outcome data from a single treatment or program cannot be 
interpreted scientifically (McLellan and Durell 1996). While convergence between multiple respondents 
creates more valid measures, often client and therapist evaluations alone provide adequate and useful 
assessments, especially when standardized measures are employed. 

The key ingredients are assessment of client outcomes and client satisfaction. The outcome reports 
can document program performance for managers, clients, and payers. Satisfaction data can help spot 
areas where the process can be improved (Nguyen, Attkisson, and Stegner 1983). Recent news reports, 
for example, reveal an HMO responding to client dissatisfaction with appointment processes (Graham 
1995). Now the HMO offers the same or next-day appointments instead of a delayed visit. Anyone 
who calls and asks for an appointment that day will get one. "Our approach to a member who called 
before was, 'are you sure you want to be seen (by a medical provider)?' Now it's 'when do you want 
to be seen?"' This important change in the service would not have happened without client/customer 
satisfaction reports. 

Client satisfaction information, however, may not be enough. Summaries of satisfaction may not pinpoint 
what might be wrong with the health care system. By the time the information works its way back to 
front-line managers and providers, it may too general to be helpful. A client satisfaction survey may not 
help front-line professionals to provide better service or to solve problems that cross departmental or 
service boundaries. Front-line personnel often need the results of root-cause analysis (Reichheld, 1996). 
Focus groups, as an example, that converge on dissatisfied customers and those who defect from the 
system can be rich sources of information about needed adjustments in the health care delivery system- 
adjustments that may not be clearly revealed in satisfaction surveys. 

Criteria for Selecting Outcome Measures. Several authors identify the criteria 2 for selecting outcome 
measures (Attkisson et al. 1978; Ciarlo et al. 1986; Ciarlo 1982; Mirin and Namerow 1991; Vermillion and 
Pfeiffer 1993; Burlingame et al. 1995): 

• The measure should meet minimal psychometric standards including reliability, validity, 
sensitivity, nonreactivity to situations, and minimization of respondent bias. If a measure does 
not have known reliability or validity, then its use is discouraged. This requirement eliminates 
most individualized (or homemade) instruments. Internal consistency reliability (coefficient 
alpha) estimates should be at .80 or above and test-retest should exceed .70. Validity coefficients 
should be at least .50 and are preferred at .75 or above. 

• The measure should be suitable for the population under care. In managed care settings, nearly 
75% of all patients present adjustment problems, affective (anxiety or depression) problems and/ 
or problems with daily living (Ludden and Mandell 1993). Human service measures should tap 
symptomatic and psychosocial functions of the client (Russo et al. 1996). 

• The measure should be easy to use, score and interpret. While some human service literature 
on outcomes suggests multiple instruments (Waskow and Parloff 1975), practice seems to follow 
a more simple approach (Lambert and Hill 1993). Simple methodology and procedures insure 
uniformity (Ciarlo et al. 1986). To guarantee outcome assessments are integrated into human 
service practice, brief and understandable instruments can report client status simply and 
objectively. If a measure is used frequently and addresses key dimensions of presenting problems 


2 For an advanced discussion of measurement, measurement error, reliability and validity, the reader is referred to Bohrnstedt, GW (1983). 
"Measurement." In Handbook of Survey Research edited by Rossi, PH, Wright, JD and Anderson, AB. San Diego: Academic Press, Inc. 
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and/or relates to treatment goals, then it becomes an easy addition to the clinical record. It can 
also reduce the effort spent on progress notes. 

• The measure should be relatively low cost. If many clients are to be assessed regularly, then 
expensive instruments will present prohibitive demands on limited resources. Impossible 
requests for time and money are likely to result in no evaluation at all. 

• The measure should be useful in clinical service functions and for evaluation purposes. The 
measure should be useful in planning treatment, measuring its impact and predicting outcome 
(American Psychiatric Association 1994). The measures should reflect meaningful change. Some 
scales mix broad improvements in symptomatic and functional areas. Others attempt to separate 
symptom distress, interpersonal relations, and social role performance (Lambert, Lunnen and 
Umpress 1994). Sometimes a measure is not used for clinical decisions about individualized client 
changes, but it is helpful in assessing how groups of clients perform. The aggregated analysis can 
be powerful in assessing program effectiveness and in documenting client progress to clients, 
clinicians, program managers, payers and legislative or regulative groups. 

While only exploratory solutions are offered on what are good outcome measurements, human service 
programs must carefully select from available measures to survive the descending mantle of managed 
care enveloping all health care programs. The struggle is to balance sound research methods with the 
demands of a fast-paced market-driven business (Freeman and Trabin, 1994). Ciarlo (1996) suggests 
outcome for managed human service care in frontier rural areas should focus on one (or more) of the 
following type's outcome assessment for: 

• Adults using general measures such as global assessment of functioning (GAF), a role functioning 
scale (RFS) or a composite score from a symptom check-list (SCL-90-R or BSI) or a combination 
of behavior and symptom identification scale (BASIS-32) or the MOS 36-item short-form health 
survey (SF-36) 

• Children and adolescents using a behavioral and symptom checklist oriented to younger clients 
(Children Behavioral Checklist or CBCL) since adult scales are usually inappropriate or ineffective 
for children and adolescents, seriously and persistently mentally ill (SPMI) people focusing on the 
lower end of the functioning continuum relative to meeting basic needs, securing self-support via 
employment, and avoiding inappropriate and/or violent behavior functioning continuum relative 
to meeting basic needs, securing self-support via employment, and avoiding inappropriate and/ 
or violent behavior. 

• The GAF scale is recognized in DSM-IV (APA 1994), but not in DSM-V (APA 2013). In DSM-V the 
multiaxial system was removed, thus leading to the recommended deletion of GAF. However, 

a measure of global disability from the World Health Organization (WHO), namely, WHODAS 
2.0, has been included in the assessment measures section. The WHODAS 2.0 domains assess 
cognition, mobility, self-care, getting along with people, life activities and participations. It has 
36 item and 12 item versions that may be administered by the client (self), the interviewer or 
by proxy. (The GAF scale is retained because of its current use and its role as an understandable 
measure in cost-outcome and cost-effectiveness illustrated later.) 

• SAMHSA recommends using the MHISP 28 item satisfaction survey for adults as part of 
their Uniform Reporting System. Currently 52 states and territories are using this instrument 
(Lutterman 2013). SAMSHA also recommends using the Youth Services Survey-Family (YSS-F) for 
youth. 42 states currently use this instrument (Lutterman 2013). 

• Alcohol and other substance abuse identifying the special impairment arising from alcohol and 
drug abuse. Table 2: Selected Program or Service Outcome Measures reviews 12 measures 
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including a client satisfaction scale. The measures, which 
tend to be inexpensive, are assessed for reliability, 
validity and the ability to produce an overall score that 
can be linked to costs. Samples of the instruments can 
be obtained from the authors, sponsors or through the 
Health and Psychosocial Instruments (HAPI) database. 3 
Key work of the primary authors or sponsors is included 
in the references. In an independent and separate 
research effort, Sederer and Dickey (1996) concurrently 
review 10 of the 12 suggested measures. 

Costs, outcomes and effectiveness 

With increased accountability, service providers of all sizes are 
being asked to demonstrate their effectiveness with outcome 
data. Outcome data can provide valuable information for 
accountability and for the improvement of clinical services 
and programs (Newman and Sorensen, 1985). Demonstrating 
effectiveness by itself, however, is usually insufficient. In 
managed care settings, effectiveness must be linked with costs. 

Callahan (1994) suggests outcomes provide a method for 
evaluating the cost-effectiveness of services. Her approach 
involves outcomes, effectiveness and cost-effectiveness as 
evidenced by the questions for varying stakeholders: 

• Client: 

• How does my progress and length of service compare to the progress made by other persons 
with similar characteristics? 

• Have my symptoms improved (or changed) as reflected by a valid scale or assessment tool? 

• Human service Staff: 

How does the progress of this person compare to the progress of my other clients with 
similar characteristics? 

• Have the client's symptoms improved as reflected by a valid scale or assessment tool? 

• Program Manager: 

• What was the rate of effectiveness for each type of service and treatment alternative? 

How many clients were served? At what cost? 

How does our program compare to others with similar services? 

• Policy Maker: 

• What types of service utilization patterns have the best (most effective) outcomes for specific 
types of clients? 

• Are these outcomes being achieved in the most cost effective manner? 


Are Social Detox Programs Safe? 

Are social model detoxification programs 
safe and adequate for treating persons 
with alcohol withdrawal symptoms ...? 

The author says "yes: The majority of 
alcoholics can be detoxified safely in 
social model programs which present 
two main benefits: cost-efficiency and 
patient's increased commitment to 
treatment (compared with the patients 
treated at medical model programs). 
Medically operated detoxification programs 
appeared necessary for patients with a 
severe withdrawal conditions at intake 
(abnormal blood pressure and pulse) and 
those with a history of severe withdrawal 
symptomatology. Screening at intake 
is critical to ensure the safety of the 
patient and the appropriateness of the 
detoxification program". Beshai, N.N. 
(1990), September-October. Public-Health 
Reports, 105, 475-481. 


3 The database can be contacted at Behavioral Measurement Database Services, P.O. Box 110287, Pittsburgh, PA 15232-0787; telephone 
412.687.6850 jor fax 412.687.5213. 
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4 The MHISIP Report Card (Customer Survey) is only part of a larger set of instruments; reliability and validity are not specified until more evidence is collected. 

5 The MHISIP Report Card (Customer Survey) did not propose cost-outcome analysis, but the logical basis (subscale scores) does exist. 

6 The Youth Services Survey-Family is completed by parents. 
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The client and human service staff questions use outcomes (or comparative outcomes) to assess 
effectiveness. 4 The client is asking, "Am I getting better?" as a measure of progress or effectiveness while 
the clinician is stating, "Are my clients improving, especially when compared to a relevant comparison 
group?" When the program manager and policy maker frame their questions, they are asking 
comparative cost-outcome or cost-effectiveness questions. "How do my costs and outcomes compare to 
other programs?" and "Are the outcomes most cost-effective" requires comparative costs and outcomes 
to assess cost-effectiveness. 5 

Outcomes in all behavioral sciences now face additional requirements to be documented by evidence- 
based practice (Torrey et al. 2012) and practice based quality improvements (Gramshaw et al. 2006). As 
an example, psychosocial treatments for ethic minority youth call for evidence-based practices (Huey, et. 
al. 2008). In a similar way, parenting and family support strategies are expected to be based on evidence- 
based practices (Turner and Sanders 2006) Social work, as another example, has developed strategies for 
evidence-based practice (Bond et al. 2009) 

Cost-Outcome and Cost-Effectiveness. Cost-outcome assessment (tying cost to clinical outcome) is one 
key to building viable cost-effectiveness analyses for program evaluation and accountability (Newman 
and Sorensen, 1985). Figure 1 identifies the major financial, statistical and evaluation tasks required for 
cost-outcome and cost-effectiveness analysis. 

Starting with total costs of a (public) human service organization, costs are refined to the per unit cost 
of service. Statistical data on professional staff activities are required to assign personnel costs, while 
information about services (e.g., units of service) is necessary to unitize program and service costs. With 
unitized costs of service and accumulated services received by specific target groups, total costs for an 
episode of care may be computed. Evaluation tasks then involve the selection of a target group, pre- 
intervention assessment, and careful non-experimental assignment of clients to varied treatments or 
services. Random assignment is ideal, but practical constraints argue for quasi-experimental procedures 
which try to equate for problem severity and other key characteristics of clients. After post- intervention 
measurements, outcomes are assessed. Then costs are related to outcomes for the final cost-outcome 
report. If cost outcomes are calculated on more than one service and comparatively analyzed, cost- 
effectiveness can be assessed for optional approaches for specific target groups (Thornton et al. 1990). 

Illustrative example of cost-outcome and cost-effectiveness. As measures of human service 
accountability and program management, cost-outcome and cost-effectiveness are interrelated. Cost- 


4 Statistical assessments of outcomes can be a complex issue. Simple gain scores (viz., time 1- time 2) are subject to much deserved 
criticism. If pre- and post-scores are correlated at reasonable levels (e.g., .3 to .4) and are linear, then analysis of co-variance (ANCOVA) with 
time 1 as the covariate may be explored. The results have to be interpreted with caution, however, since those with higher initial scores 
can be expected to improve at a higher rate than those with lower scores. By relating the actual gain to a potential gain and analyzing 

the percentages with ANCOVA is somewhat more defensible. The analysis uses the form: Time 2-Time 1/ Ideal-Time 1= %. ANCOVA is 
problematic in any event. First, the statistical assumption that the treatment and the covariate do not interact systematically is not met 
since entry levels of a mental health condition (e.g., depression) and treatment approaches do have a systematic interaction. Second, 
since the interaction of treatment and entry level is of concern along with the main effect of treatment, any statistical control procedure 
to partition or subtract out information typically used in clinical decision-making should be viewed with caution. Analysis of variance with 
repeated measures poses similar problems. 

5 Most parametric statistical analyses pose problems in comparing the effectiveness of two approaches to mental health treatment. The 0 
(Theta) technique (with a x2 statistic) can analyze two outcome matrices by comparing the two approaches against an ideal matrix. The test 
is sensitive to the magnitude of differences in treatment effects and represents a measure of the differences in patterns of client outcomes 
for two treatments at measured levels of intake functioning ... relative to a hypothesized pattern of outcomes. See Newman and Sorensen 
1985 and Ross and Klein 1979. Other approaches include structural equations that are beyond the scope of this paper. 
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outcome analysis find s the programmatic resources consumed to achieve a change in a relative measure 
of client outcome (e.g., functioning). Cost-effectiveness analysis compares beneficial program outcomes 
to the cost of programs (or modalities or techniques) to identify the most effective programs. The 
following example illustrates the basic steps. The outcome measures used in the illustration identifies 
the major criteria for client performance (Figure 2) and the scale metrics (Figure 3). The scale is a global 
assessment of the four criteria scaled into nine levels of measurement. Levels 1 to 4 are considered 
dysfunctional while levels 5 to 9 are deemed functional. Figure 4 is a basic cost-outcome matrix using 
only the dysfunctional-functional level of functioning. Level of functioning is assessed at the start and 
end of a time period for a specific target group of clients. Combining the two rows and two columns 
results in four-cells: 


cell a: 
cell b: 
cell c: 
cell d: 


start: dysfunctional (1-4 ratings) 
start: dysfunctional (1-4 ratings) 
start: functional (5-9 ratings) 
start: functional (5-9 ratings) 


end: dysfunctional (1-4 ratings) 
end: functional (5-9 ratings) 
end: functional (5-9 ratings) 
end: dysfunctional (1-4 ratings) 


Figure 1. Overview of Major Tasks in Cost-Outcome and Cost-Effectiveness Studies in 
Human Service Organizations 



Source: Sorensen, Hanbery, and Kucic, 1983. 
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Figure 2. Major Criteria for Performance 

1. Personal self-care (adjust to age level). 

2. Social functioning (adjust to age level). 

3. Vocation and/or educational functioning 

a. Working adults. 

b. Homemakers and/or parents and/or elderly. 

4. Evidence of emotional stability and stress tolerance. 


Figure 3. Develop Scale Metrics 


Level 

Functionality 

1 

Dysfunctional in all four areas. 

2 

Not working; intolerable; minimal self-care; requires restrictive setting. 

3 

Not working; strain on others; movement in community restricted. 

4 

Probably not working, but may if in protective setting; can care for self; can interact but avoid 
stressful situations. 



5 

Working or schooling, but low stability and stress tolerance; barely able to hold on and needs 
therapeutic intervention. 

6 

Vocational/educational stabilized because of direct therapeutic intervention; symptoms 
noticeable to client and others. 

7 

Vocational/educational functioning acceptable; therapy needed. 

8 

Functioning well in all areas; may need periodic services (e.g., med check). 

9 

Functioning well in all areas and no contact with behavioral health services is recommended. 


Figure 4. Cost-Outcome Matrix (basic) 



END 3/31/X 


Dysfunctional 

Functional 

START 1/1/X 

Cell A: 

Cell B: 

Dysfunctional 

$ of Services 

$ of Services 

1-4 

n 

n 


X-bar; sd 

X-bar; sd 


Cell C: 

Cell D: 

Functional 

$ of Services 

$ of Services 

5-9 

n 

n 


X-bar; sd 

X-bar; sd 


n = xxxx 
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Next, for the clients in each cell, the services received and 
related unit-of-service costs are multiplied and summed and 
statistics such as the mean (x-bar) and standard deviation 
(sd) are computed for each cell. Of special concern is cell c 
since moving from functional to dysfunctional may suggest 
clinical risk. Cell a is of interest since the clients have not 
moved from a dysfunctional status and often represent 
high consumption of expensive services. Cell b is of interest 
since the clients moved from a dysfunctional to a functional 
level. Finally cell d may deserve a review to assess resource 
consumption by clients who started and ended the review 
period as functional. 

Figure 5 is an expanded matrix of costs and outcomes using 
all nine points of the scale developed in Figure 3. Individuals 
starting and ending at the same level are on the diagonal 
while those showing improvement are above the diagonal 
and those showing regression are below the diagonal. 

Means and standard deviations are computed for each cell. 

Client change and costs are aggregated by improvement, 
maintenance, and regression (as shown conceptually 
in Figure 6) and illustrated with sample values in Figure 
7. Client outcome (e.g., improvement, maintenance or 
regression) and the resources used to achieve the outcome 
are linked in Figure 7. Note in the illustration that 40% are 
improved (with 19% of the resources), 50% are maintained 
(by consuming 71% of the resources) and 10% regressed 
(while receiving 10% of the resources). 

In cost-outcome analysis, there is no way to document 
whether change during service is actually caused by the 
intervention or is simply concurrent with it. Gathering 
comparative cost-outcomes on optional services (e.g., A 
vs. B) may separate the effects of service strategy and cost 
differences. Potential intervening variables, such as history, 
selection bias, practice effects, maturation and other factors 
unrelated to the service can be controlled by random 
assignment to alternative services or by less desirable 
quasi-experimental methods such as matched comparisons. 

The purpose of the analysis is to reach conclusions about 
the relative cost and effectiveness of the services. Figure 8 
reviews the logical relationships and choice points about 
two services (A and B). Seven of the choice points are self-explanatory (e.g., A is as effective and A costs 
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Saving Youth and Your Money Too? 

Programs like the Community Intensive 
Treatment for Youth (CITY) are based on an 
assessment that includes academic diagnostic 
testing, a home visit, completion of parent 
and youth data questionnaires, collection of 
school data, and observation of behavior. The 
treatment plans begins with the Youth's goals 
and consists of four or more parts, including 
academic, behavioral, family and group 
components. For each part a reasonable goal is 
set. A plan to reach the objective and a means 
of evaluating the effectiveness of the plan are 
developed. Overall CITY program objectives 
are to identify each juvenile's strengths and 
weaknesses, to provide an individualized 
environment in which a juvenile can develop 
the skills necessary for successful living, and to 
alter the natural environment of the juvenile so 
that the newly acquired skills are fostered and 
previous negative behaviors are discouraged. 
Tracking of 231 juveniles for 1 year after they 
exited seven of the programs showed that 72 
percent had no new adjudications. The cost per 
person per day for the 240 CITY program slots 
in eight program locations is $43.83 and the 
cost per bed per day in the juvenile institution 
is estimated to be more than $120. Earnest, D. 
(1996) Corrections Today. 58, 70-73. 

Research questions: Are the juveniles in the 
juvenile institution comparable to the juveniles 
in the CITY program? How were the subjects 
assigned to the CITY program ? What is the 
total cost of an episode of care in the two 
settings ? (Unit costs may differ, but how many 
units of service did the two populations receive 
so a total cost can be estimated.) 
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Figure 5. Cost-Outcome Matrix (expanded) 
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Figure 6. Cost-Outcome Matrix (summary) 
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Figure 7. Cost-Outcome Matrix (table) 


Client 

Change 

N 

(000) % 

Total $ 

(000) % 

X-Bar 

IMPROVE 

4 

40% 

$1,200 

19% 

$300 

MAINTAIN 

5 

50% 

$4,500 

71% 

$900 

REGRESS 

1 

10% 

$600 

10% 

$600 


10 

100% 

$6,300 

100% 

$630 


Figure 8. Cost-Effectiveness Matrix 



COST: A vs. B 

EFFECT: A vs. B 

A < B 

A = B 

A > B 

A < B 

? 

B 

B 

CO 

II 

< 

A 

A or B 

B 

A > B 

A 

A 

? 


less, therefore choose A) while the cells with question marks (?) are not clear conclusions (e.g., A is less 
effective and A costs less) 

Effect of capitation. Cost-effective care with limited resources can be reinforced by capitation (Lehman 
1987). The Monroe-Livingston demonstration project, as an illustration, evaluated capitated funding of 
human service care in contrast to fee-for-service in a seriously mentally ill population. After a two-year 
follow-up, Cole et al. (1994) found patients in the capitation had fewer hospital inpatients days than the 
fee-for-service group, while both groups were similar in their functioning and level of symptoms. This 
report evaluated effectiveness using outcomes. Reed et al (1994) evaluated total costs and benefits in 
the same demonstration and concluded, "... capitation funding can promote care of seriously mentally 
ill persons in community settings at lower overall costs." This report then linked costs to outcomes to 
assess cost-effectiveness. 

Human service programs need to document costs and outcomes at a minimum. Armed with cost and 
outcome data, a cost-outcome report is possible. Medicaid (and Medicare) purchasing authorities, state 
human service authorities, managed care vendors, HMOs and business coalitions are likely to respond 
positively to cost-outcome information. Cost-outcome can also continuously assess, plan and improve 
services. Where comparative cost-outcome information is available, cost-effectiveness reports may be 
possible, but in human service environments these opportunities may be limited. Cost-effectiveness as 
a strategy for the design and deployment of human service services is reflected in several applications 
reviewed or proposed. In some instances, highly acceptable approaches (in theory) must by tempered by 
the realities faced in deployment (in practice.) 
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Should you read this chapter? 

It is tempting to skip a chapter on financial management because the topic smacks of dollars and cents 
and not of more attractive and exciting issues such as clients, services and treatment outcomes. The 
temptation may be fatal. No organization (human services or otherwise) survives successfully without 
effective financial management. Great ideas about human services must work out financially or they 
are not likely to work out at all. You do not have to be a financial wizard to understand the basic tools 
of financial management - tools that ultimately govern your ability to acquire, allocate and account for 
resources. 

Are you in the target audience? 

You are the target audience if you deliver human services (e.g., human service program or clinical 
director), or if you fund or evaluate human services (e.g., a county human service administrator or a 
funding agency at any level of government) or if you want to learn about human services. 

Why is this chapter relevant to you? 

Financial constraints and opportunities emerge from: 

• Shifting funding sources (e.g., growth of Medicaid expenditures with cutbacks in federal funds to 
states and local governments) 

• Increasing demands for more organized services (e.g., managed behavioral health care) 

• Reorganizing responsibility for the delivery of human services (e.g., deinstitutionalization and 
emergence of community based programs) 

• Developing new service systems (e.g., shift from provider-centered to client-centered services 
and assertive community treatment [ACT]) 

• Increasing demands for linkages of services with client outcomes and client (family) satisfaction. 

Coping with these opportunities and constraints requires a fundamental understanding of the financial 
management of a human service system (Sorensen, 1989, 1992 [2], 2000). 

What should you be able to do after reading this chapter? 

You will be able to assess the strategic management process, comment on the approach to Total 
Quality Management (TQM), and evaluate the relationship of the strategic and operational financial 
management system to the overall management process. You will be able to review the key performance 
indicator system and employ the foundations of financial planning and control. You will be able to 
evaluate costs, budgeting, and cost-outcome and cost-effectiveness. In brief, you will gain a set of fiscal 
survival skills. 

Planning and Performance Information 

Management of a human service organization requires plans that relate to organizational goals and 
objectives based on the relative benefits and costs of optional courses of action while maintaining 
sufficient control to ensure efficiency and effectiveness in pursuing the organization's mission. 

Planning information reveals what services the organization will render and to whom and what resources 
the organization will use to provide these services. Performance information measures how effectively 
the organization is doing its job and how efficiently the organization is using its resources. 
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Financial Management 

Managers of human service organizations must acquire and use resources to create effective human 
services at a minimum cost. As part of the general management process, financial management focuses 
on the analyses, decisions, and actions related to: 

• acquisition of resources (financing), 

• allocation of resources (distribution), and 

• accountability for the use of resources (evaluation). 

A decision, for example, to develop new programs or service requires financing. The financing decision, 
in turn, may influence (and may be influenced by) the mix of services and professional staffing. The 
rate set for services may influence use and third-party payment flows. The amount and kind of service 
actually received by each client are linked to the financing source and to decisions on which staffs 
perform what services. The priority established by a State or Federal agency for a service may or may not 
coincide with local priorities, thus slowing or hampering service expansion. Funding sources may require 
some type of outcome evaluation— what happened to the client? Clarifying the full set of relationships 
necessary to make optimal decision is difficult. 

An Illustration. 

An illustration may shed light on the complex relationships in financial management: 

Financing (or obtaining the resources): To phase up a community-based human service programs and 
to phase down psychiatric facilities requires a shift in the funding of human service systems. To achieve 
a single stream funding requires the central funding authority to channel funds to community human 
services. The community services may now have the authority and responsibility for purchasing inpatient 
services from public psychiatric facilities or from community alternatives. 

Distribution (or allocating the resources): New community services may envision home-based crisis 
stabilization, crises care home, continuous treatment teams, assertive community treatment (ACT) 
or intensive case management staffed by varied types of personnel such as a psychiatric nurse 
practitioner for continuous treatment teams or non-human services providers to administer case 
management services. Imagine if a State funding authority places caps on reimbursement (e.g., limits 
on case management rates), then the amount of service provided may be a function of available 
funding resources. A State or Federal emphasis on crisis response capabilities may be at variance with 
established local partial hospitalization or day treatment programs. 

Evaluation (or accounting for the resources): Public funding sources may require financial reports on 
how and where the resources were expended. In addition the agency may require an evaluation of the 
customer's satisfaction with the services or some estimate of how well the customer responded to the 
services (e.g., client level of functioning). 

A financial manager may examine some aspect of a decision individually and assume little impact 
elsewhere. A series of separate decisions may work at cross-purposes and produce bad results. 

Simple one-by-one financial decisions can wreck effective programs and financially unsound services 
and programs providing good client service can scuttle an entire agency. Client service and financial 
management concerns must merge if human service organizations expect to survive in the long run 
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(Sorensen [3], 2000). A long run perspective requires strategic financial management to be an integral 
component of the agency's strategic management. 

Financial Management as an Element of Strategic Management: An Overview of 
Strategic Management 

Someone in the financial management of the human services sector may wonder about the value of a 
summary of the strategic management process. Consider, for example, the following questions: 

• Is there an appropriate match between financial resources and goals? (This question evaluates if 
the organizational resources are being generated and used according to the vision and mission.) 

• Do the sources and uses of resources match appropriately? (This question examines how various 
resources are derived and consumed.) 

• Are the financial resources sustainable? (This question probes the stability of revenues, expenses, 
assets, and liabilities.) 

In brief, high-level financial management links to the strategic management process and requires a 
mutual understanding of both agendas. There is a need "to dance the waltz together." 

An integrated model of strategic management involves strategic planning and strategic control. 

Strategic planning includes formulating and evaluating optional strategies, choosing a strategy and 
developing plans for placing it into practice. These plans should reflect the values and expectations 
of the stakeholders, the organization's vision or mission, goals and objectives, external and internal 
environments of the organization and major policies. Strategic control is assuring compliance with the 
strategic plan. 

The vision statement 
declares key values 
or ideals while the 
mission statement 
defines why the 
organization exists 
and why it competes 
in the public sector. 

Goals (or the broad 
directions or results) 
and objectives 
(the focus on more 
specific targets) 
are what the 
organization wants 
to accomplish. The 

formulation of goals and objectives uses an assessment of the organization's environment and resources. 
Pursuit of organizational goals and objectives leads to the formulation of a strategy including a general 
approach down to specific plans to be implemented. Organizational policies help to limit choices among 
optional strategies while procedures guide the implementation. An illustrative and interactive model 
appears in Figure 1. 


Figure 1. Integrated Model of Strategic Management 



WHY? WHAT? HOW? WAY? 
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The starting point for the formulation of a strategy is the vision statement. As an illustration, West 
Virginia's vision is "a community that values and respects people and is responsive to their individual 
needs, wants and desires for the enrichment of their lives." (West Virginia, 2001) Vision statements are 
general, lofty and inspiring. The mission, on the other hand, is more down to what the organization tries 
to do. West Virginia states "We ensure the positive meaningful opportunities are available for persons 
with mental illness, chemical dependency, developmental disabilities and those at-risk. We provide 
support for families, providers and communities in assisting persons to achieve their potential and gain 
greater control over the direction of their future." (West Virginia, 2000) West Virginia adds an additional 
values statement: "We believe in integrity-based leadership that is flexible enough to respond to 
change that guided by consumers, employers, and their community." Note these statements emphasize 
consumer empowerment, normalization and client-provider decision-making about services to meet 
consumer needs. 

An example of a mission statement is from the Tennessee PL 99-660 Mental Health Plan (1991) 
(Yearwood, et al., 1992): The mission statement of the Department of Mental Health and Retardation 
"... is to improve the quality of life for consumers and their families as they define quality of life. ... 

The Division of Mental Health Services (DMHS) is committed to a mental health delivery system that 
is centered on the stated needs and preference of primary consumers that is accountable to these 
individuals and their families, and ... consumer empowerment through meaningful involvement by 
consumers in all aspects of planning and providing services. Programs and services supported by the 
DMHS will assist adults with severe disabling mental illness to control the symptoms of their illness, to 
develop the skills and acquire the supports and resources they need to succeed where they choose to 
live, learn, and work. Programs and service supported by the DMHS will assist children and their families 
in their efforts to remain together and to develop skills and acquire the supports and resources they 
need to function successfully in the most normalized setting possible. Services will assist adults, children, 
and their families to maintain responsibility to the greatest extent possible for setting their own goals, 
directing their own lives, development and acting responsibly as members of the community..." (p 22 
and 23). 

The foregoing mission statement is about the purpose of the agency. Once the purpose is clear, then the 
goals (viz., broad directions and intentions) and objectives (viz., more specific ends to be met) are next. 
Illustrative goals for a public mental health agency might be (Yearwood, et al., 1992): 

1. Design and deliver services guided by the needs and preferences of consumers and family 
members. 

2. Integrate inpatient treatment settings as a flexible back-up to comprehensive community support 
and other mental health services. 

3. Allocate most public health financial and human resources to community services focused on 
persons with serious mental illness. 

Objectives are to be written, understandable, challenging, attainable, measurable and dated (reference). 
Objectives for the forgoing goals (for several different states) might include: 

1. For persons with seriously and persistently mental illness, reduce state hospital usage by 

providing an assertive community treatment (ACT) team with in-home stabilization, community- 
based placements or brief inpatient treatment and admission to a state hospital as a last resort 
(Santos, 1997). 
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2. Develop cost-effective supported employment program rather than rehabilitative day treatment 
(Clark etal., 1996, 1998) 

3. Increase independent living and successful community integration by developing intensive case 
management focused on: 

a. improved housing, 

b. improved employment, 

c. increased network of friends, and 

d. decreased inpatient admissions treatment (Stroul, 1989). 

Generally goals and objectives provide targets focused on kinds of clients, types and distribution of 
services, finances, staffing, and growth (or decline). 

Assessing the strategic situation 

Opportunities and threats of the general environment are generally factors over which the organization 
has little direct control. Generally these include the economy, technology, social factors such as 
consumer preferences and political or legal issues. Funding of human services in a general national 
health insurance program, for example, may be beyond the direct influence of most human services 
programs. Another general trend beyond direct control is the shift in the desires of human services 
consumers to participate in formulation of the type and location of services. Changes and trends in these 
factors should influence the directions of the strategic plan. 

Internal analysis includes the listing of internal strengths (e.g., qualified professional staff, good 
distinctive competence, strong financial position, cost advantages, good image and reputation, 
demonstrated management, capability for innovation) and internal weaknesses (e.g., high cost 
compared other providers, weak marketing skills, poor internal information systems, unfocused strategy, 
lack of shared values). 

External opportunities and threats include the listing of potential opportunities (e.g., adding new 
programs, entering new segments or markets, rapid growth in demand, exits of rivals from the market) 
and potential threats (e.g., slower market growth, adverse governmental policies and funding, stronger 
competitive pressures, adverse economic conditions). 

Competitive position assessment 

Assessment should be a careful review of the strengths and advantages and weaknesses and 
disadvantages to evaluate if the organization's position is improving or slipping and why. Does the 
organization have a net competitive advantage or disadvantage? How strong is the competitive 
advantage or how bad the competitive disadvantage is and how hard it will be to overcome the 
disadvantage? What are the implications for overall strategy? What are the hurdles the organization will 
have to overcome in view of the competitive forces, future competitive conditions and moves of key 
rivals (University of Denver, 2001)? 

Strategy formulation, evaluation and choice 

The organization should define a root strategy, set strategic goals (e.g., five) and key performance 
objectives (e.g., three to five) and formulate a strategy to achieve the target objectives. The mission 
statement should indicate what the organization wants to accomplish and the markets it wants to serve. 
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The overall strategy should indicate the thrust of the business-level strategy is (e.g., aggressive grow-and- 
build, conservative hold-and-maintain, whether major offensive moves should be initiated and what kind 
of moves, or whether the emphasis should be on defense and fortification). The business-level strategy 
should address how to respond to changing human services industry and competitive conditions, how 
to compete and how to achieve a stronger position and what will be the resource allocation priorities 
within the organization (Sorensen, 1989). 

The strategy should develop a competitive approach that will produce a sustainable competitive 
advantage. Typical options include low cost service, differentiation of service or special service niche. The 
approach pinpoints the competitive advantages the organization should strive for and identifies the acts 
to create this advantage. Included also are specific offensive or defensive moves needed as part of the 
overall competitive approach. 

Implementation of the strategic plan comes from the organizational functions of client services, 
programs and administration. The analysis of the proposed financial strategy requires a determination 
if all the functional strategies can be funded at reasonable level of costs. Note the definition of client 
services, programs and administration shift as the organization shifts from a direct health provider to an 
administrative unit at a city, county, state or federal level. 

Finally, a review of the strategy should assess if it meets all the industry and organizational strategic 
issues identified earlier. Recommendations include both short-term and long-term priorities. 

Strategic financial plans 

Most strategic financial plans include a clear statement of the objectives of the entity along with the 
costs and financing of the objectives over a specified time period such three years, five years and in 
the case of large state agencies, twenty-years. The format presented in Figure 2 illustrates the basic 
approach. Specifying objectives, programs, and services requires the output of the strategic planning 
process. Without careful strategic planning, the strategic financial plan cannot take shape. 


Figure 2. Strategic Financial Plans 


Variable 

T 

T+l 

T+2 

T+5 

Objective/Goal 
1. AAA 

Expenses 

$xxx 

$xxx 

$xxx 

$xxx 

2. BBB 

$xxx 

$xxx 

$xxx 

$xxx 

Total Expenses 

$xxx 

$xxx 

$xxx 

$xxx 

Funding 
1. YYY 

Revenues 

$xxx 

$xxx 

$xxx 

$xxx 

2. ZZZ 

$xxx 

$xxx 

$xxx 

$xxx 

Total Revenue 

$xxx 

$xxx 

$xxx 

$xxx 

Net Over (Under) Recovery 





(Total Revenue- Total Expenses) 

$xxx 

$xxx 

$xxx 

$xxx 

Beginning net assets (or fund balance) 

+ $xxx 

$xxx 

$xxx 

$xxx 

Ending net assets (or ending fund balance) 

= $xxx 

$xxx 

$xxx 

$xxx 
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At this point, reviews and reevaluation of the costs and funding of objectives are possible. If the strategic 
financial plan is a "rolling" plan (say, a five year plan) at the end of each year, the current year is dropped 
and a new fifth year is added. New information about objectives, programs and services is added or 
revised. If, for example, new funding sources appear (or disappear) the information is included in the 
plan. If the objectives change (e.g., increase, decrease, elimination), the appropriate revenue and 
expense information are included. The power of the strategic financial plan flows from anticipating what 
is and what might be happening to the organization. If multiple conditions are anticipated, then multiple 
runs and outputs may be required to provide top managers with an adequate view of the potential 
future. 

Financial management as an element of Total Quality Management (TQM) 

Several of the key forces driving Total Quality Management focus on the: 

• Intensifying competition for human services resources (e.g., pressures to fund other health 
services or other social services such as education) 

• Increasing awareness of the inefficiency in the delivery of mental services (e.g., home-based 
crisis stabilization is more efficient than inpatient hospitalization) 

• The emergence of the new understanding high quality is possible along with lower costs (e.g., 
intensive case management can improve the quality of care to a target client group while 
lowering costs, especially inpatient admissions treatment) 

• New knowledge about the needs and capabilities of individuals with severe and persistent 
psychiatric disabilities (e.g., many individuals with severe mental illness and emotional 
disturbance can maintain jobs, friendships and families) 

From a practical managerial viewpoint, quality is ongoing improvement. The improvement can be 
incremental (e.g., a modest change) or breakthrough (e.g., a radical redesign of the delivery of human 
services). 

Information Systems and Key Performance Indicators (KP1) 

A significant cornerstone in any approach to financial management is the organizational information 
system (IS). The information system should produce information that (Chapman, 1976): 

• Assess the patterns of service delivery (e.g., who receives how much of what type of services 
when and where)? 

• Define how current resources are being acquired and consumed (e.g., what are the major 
sources of revenue and how are they being spent)? 

• Provide monitoring aids for various health care providers and managers (e.g., are the admissions 
appropriate)? 

• Develop data for multiple reporting requirements (e.g., reporting to funding agencies) 

• Create a data base for planning (e.g., monitor changing demographics to formulate future 
program planning) 

• Assess the outcomes of services provided (e.g., level of client functioning; impact of a program 
on a community) 
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.Significance of Key Performance Indicators in financial management 

Enhanced financial management results if the outputs of an information system (IS) focus on key 
variables or key performance indicators (also known as key result areas or key success factors). A similar 
model is in the Governmental Accounting Standards Board (GASB) service efforts and accomplishments 
reporting or SEA indicators (Bailey, 2002). Identifying key variables in a human services organization 
requires a thorough understanding of programmatics and economics of human services. Experienced 
managers are likely to give primary attention to key performance indicators. Generally these indicators 
are important to the success of the organization, are robust summaries of more complex relationships, 
are factors requiring managerial action when significant change occurs and may be sensitive to quick or 
volatile changes. 

One approach is to design the KPI around four mixes: revenues, clients, staff, and services (Sorensen, 
Zelman, Hanbery and Kucic, 1987). Illustrative ratios include: 

Revenue mixes: 

1. % distribution of revenue Revenue by source 

by source: = 

Total annual revenue 

2. Expense convergence Total standard charges 

rate by [defined] by [defined] program 

program element: = 

Total expenses by [defined] 
program element 

Ratio #1 provides insight into the revenue stream and identifies the sources of revenue of the 
organization. Ratio #2 examines the convergence of the expenses of a [defined] program element to the 
value of the services provided. The numerator is the standard charge (or fair market value) of a service 
multiplied by the number of the services rendered (thus producing the "total standard charges") while 
the denominator is the total resource (or expense) consumed by the [defined] program element. Under 
ideal conditions, the ratio would be one (1), thus suggesting the value of the service (as measured 
by what is charged for it in the market place) is equal to the resources consumed (as measured by its 
expenses). A quick application of the ratio can find "leaky boats" or program elements that are not 
carrying their own weight. If program cuts have to be made, the ratio identifies more rational candidates. 

Client mixes: 

Client health status # of severely mentally 

by [defined] age ill in [defined] age group 

group = 

Totally severely mentally ill 

This health status ratio indicates the attention an agency may be directing to varying age groups. For 
example, are children and geriatrics present? Are they underrepresented? Does the case finding of 
geriatric clients deserve more effort? If the denominator comparison was, say, total admissions, a state 
agency may identify appropriate or inappropriate admission patterns by the service providers it funds. 


56 


Human Services Program Evaluation 

Financial Management in Human Services Organizations 


Chapter 5 


Staff mixes: 

% distribution of 
clinical staff effort 
by [defined] program 
element 


Total # of clinical staff 
hours devoted to [defined] 
program element 


Total # clinical staff hours 
devoted to all program elements 


This staff ratio reveals the relative distribution of staff effort to [defined] program elements. If the 
numerator controls for the type of discipline, then the ratio indicates the distribution of disciplines 
among the program elements. 


Service mixes: 

1. Average # of service 
units per FTE day 
by [defined] program = 
element 


2. Cost per unit of 
service by [defined] 
program element 


Total # of units of [defined] 
program element 


Total FTE days by [defined] 
program element 

Direct + Indirect + Allocated 
costs of [defined] program 
element 


Total units of service of [defined] 
program element 

Ratio # 1 measures the productivity of staff in the varying [defined] program elements. While only 
measuring how hard the staff is working, it may be useful in comparative assessments to ascertain if the 
staff is being adequately utilized. A modification of the ratio is to divide the number of clients attending 
per day by the number of FTE staff to identify the average attending per day per FTE. In many of the 
newly emerging programs (e.g., crisis teams), the latter measure is a more meaningful measure. 

Ratio #2 is an overall index of resources consumed to production provided for a [defined] program 
element. Cost per unit can be a useful index of efficiency when compared over time within an agency 
or when compared to other similar organizations. In some cases the total cost index may vary between 
different sized organizations because of the amount and impact of fixed costs being spread over a 
smaller or larger number of units of service. Often the cost-per-unit is the basis for funding negotiations 
and reimbursement agreements (e.g., Medicaid) 

The essence of financial control is the comparison of planned (e.g., budgeted) level of activity with 
what is actually happening. All the foregoing KPI measures lend themselves to a "budget vs. actual" 
format. For example, if the planned unit cost of service is compared to the actual unit cost of service, the 
resulting ratio becomes a simplified tool to evaluate the performance of the organization and to detect 
when appropriate managerial investigations should occur (e.g., the ratio is markedly less or greater than 
one, thus suggestion the process is out of control). 
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Management Accounting 

Most human service managers face two difficult questions: 

• Are we doing what we should be doing? 

• How well are we doing what we do? 

Today's complex human services environment gives neither easy nor clear cut answers to these 
questions. The first question stitches back to our strategic plans and how those translated into a strategic 
financial plan. Long-term objectives and translation of those objectives into operational financial plans 
require budgeting tools. 

Describing how well the organization is functioning is somewhat easier to assess than whether the 
objectives are appropriate. Efficiency is the accomplishment of objectives as a minimum cost while 
effectiveness measures how well the objectives are achieved. Cost behavior and cost classification, 
contribution margin analysis, differential analysis, flexible budgeting and budget variance analysis are the 
management accounting tools to address the second question. 


Cost behavior 


Cost 


Variable 



Volume 


Fixed 


Step Variable $ 


Mixed 


Total cost 
at varying 
volumes 


$ _ 


Per Unit 


Management accounting examines cost behavior in relation to volume of activity. As volume of activity 
varies, a cost may increase proportionally to volume (a variable cost), may not change as volume 
changes (a fixed cost) or may 

change in stepwise fashion (a step- Figure 3. Cost Behavior 
variable cost) as volume changes. 

Figure 3 portrays these basic cost 
behaviors including a mixed set of 
costs. Copying costs, for example, 
may vary directly with the number 
of copies produced (e.g., a variable 
cost) while annual lease payments 
may remain constant (e.g., a fixed 
cost) regardless of client volume. 

Administrative assistants may 
increase as the volume of activity 
(e.g., number of contracts) changes 
and the cost behaves in a stair-step 
fashion (e.g., step-variable cost) 
since each assistant can handle a 
certain number of contracts before 
another assistant is required. 


Total 


Proportional 
to change 
in volume 


Same over 
wide range 


Total cost is a 
"stairstep" 


Constant 


Varies 

inversely with 
volume 


Varies 

inversely with 
volume 


Average total 
cost per unit 
at varying 
volumes 


Revenue behavior 


Management accounting also classifies revenues like costs. Revenues may be variable (e.g., a unit of 
service provides a specified amount of revenue) or revenues may be fixed (e.g., categorical funding for a 
service without regard to volume of service). 
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Breakeven and beyond 

If a manager knows both cost and revenue behavior, s/he can calculate a "breakeven" point that is where 
revenues and costs are equal. Basically, revenues are reduced by variable expenses to produce the 
contribution margin. The contribution margin percentage is the contribution margin 

divided by the revenue. When the fixed expenses are divided by the contribution margin percentage, the 
result is the level of revenue required to cover both variable and fixed costs. 

Assume, for example, revenues are $1,000,000 and variable costs are $400,000. The contribution 
margin is $600,000 ($1,000,000- $400,000 = $600,000) and the contribution margin percentage is 60% 
($600,000 /$1, 000, 000). For each dollar of revenue received, 60 cents goes to cover fixed expenses 
and, after all fixed expenses are recovered, the organization over-recovers its costs at 60 cents on the 
dollar. If the fixed expenses are, say, $600,000, then the required revenue to "breakeven" is $1,000,000 
($600,000/.60=$l,000,000). If the fixed expenses are $660,000, then the organization needs $1,100,000 
($660,000/.60 = $1,100,000). If the fixed expenses are, on the other hand, $540,000, then the required 
revenue is only $900,000 ($540, 000/. 60 = $900,000). If the organization receives $1,000,000 with 
$540,000 of fixed costs, the overrecovery is $60,000 ($1,000,000 less variable costs at .40 x $1,000,000 
equals contribution margin of $600,000 less fixed costs of $540,000 equals $60,000). A more direct 
computation is to multiply the revenue in excess of breakeven revenue times the contribution margin 
percentage ($l,000,000-$900,000 x .6 = $60,000). 

The analysis extends to multiple services, programs and agencies. (See a management accounting text 
for extended applications.) 

Often providers are required to do more with less. A reference (Sorensen, Vela, and Grant, 2010) outlines 
a comprehensive approach for managing revenues, expenses, policies, and practices when a provider 
organization is faced with reduced resources or increased demand for services, or both. 

Cost distinctions for planning and control 

Fixed costs are either committed or discretionary. Committed fixed costs may be fundamental (e.g., 
property taxes, bond interest payments, key personnel) and reflect long-run capacity needs. Typically 
these costs are not responsive to short run variations in activity. Discretionary fixed costs, on the on the 
other hand, are periodic costs influenced by top management decisions (e.g., scheduled maintenance, 
staff training, professional travel) and often bear little relationship to volume of activity. In times of 
hardship, these costs are subject to reduction although long-term effects may be negative. 

Variable costs are either engineered or discretionary categories. Engineered costs represent a defined 
cost to produce a given service or product (e.g., cost of a psychologist to perform testing). Shifts in 
engineered costs change the resulting service or product (e.g., moving from a Ph.D. to a Masters level 
psychologist). Discretionary variable costs represent managerial choices that may be altered without a 
fundamental effect on the service or product (e.g., switching from a brand name drug to a generic type). 

When management is looking for short-term cost reductions, discretionary variable and discretionary 
fixed costs become prime targets. Often these costs are subject to reduction without immediate adverse 
effects, but, some, if postponed indefinitely, will produce adverse effects (e.g., training or maintenance). 
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Differential costs and revenues 

Often short-term decision-making asks what the differential effects on costs and revenues are if a 
program or service is added or dropped. Only the revenues or costs that are expected to change are 
relevant to the decision. The differential revenues and costs correctly identify the financial effects of a 
decision. For example, should program B be dropped since it appears to lose $100,000 (all $ amounts in 
the analysis below are in thousands)? 



A 

B 

TOTALS 

+ Revenue 

$1,000 

$3,000 

$3,000 

- Variable Expense 

-600 

- 2,000 

- 2,600 

= Contribution Margin 

= 400 

= 1,000 

= 1,400 

- Fixed Expense 

-300 

- 1,100 

- 1,400 

= Net Over (Under) Recovery 

= 100 

= (100) 

= 0 


The differential analysis identifies the differences without B as the loss of contribution margin and the 
continuation of fixed expenses. Dropping program B results in the loss of $1,000,000 of contribution 
margin and could leave a substantial fixed expense (up to $1,100,000). Instead of "breaking even" as 
shown in the total column, the agency could face a loss up to $1,000,000 ($100,000 over-recovery from 
A reduced by the unavoidable fixed expense of B of $1,100,000). In this instance, dropping the service 
increases the losses of the organization and, therefore, is not the correct decision. 

Budgeting 

Successful human service organizations characterize good operating performance. The corner stone of 
success is sound financial planning (Sorensen, Hanbery, & Kucic, 1983). The budget process requires 
planning and encourages better administration. The master budget integrates all the organization's 
functions into an overall plan and can highlight operating and financial problems early enough for 
effective remedial action. 

Defined. The master budget is a plan of financial operations that provides a basis for the planning, 
controlling, and evaluation of activities of a human service organization. The budget is a quantitative 
expression of management's plans. The master budget summarizes the objectives of the organization's 
subunits and quantifies the activities for the next operating cycle (e.g., number of contracts, dollars of 
funding, levels of client service, number of professional staff, operating expenses) 

Human factors. Top management's view of the budget process influences the attitudes of middle-level 
managers. If top managers do not give wholehearted support, the budgeting process is not likely to be 
popular and may be the source of negative attitudes. The budget can help establish goals and objectives 
in quantitative form, direct behavior, measure results and identify areas that need management 
attention. Managers who are expected to use the budget are to be involved in its development. 
Lukewarm support from the top for a budget process can result in low levels of communication between 
managers and with only token participation by line managers the budgeting process fails. Budget failures 
make the staff wary of efforts to improve managerial effectiveness and efficiency. Budgets can be 
effective when managers understand and use them, not just execute the mechanics of the process. 
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Advantages. Skeptics are quick to point out the uncertainties and hard work of budgeting. Ignorance 
about budgeting may cause negative attitudes; sometime skepticism disguises an unfortunate 
experience or a lack of technical skills or poor motivation. The benefits of budgeting usually outweigh 
the costs because budgeting forces managers to think ahead— to anticipate and prepare for changing 
conditions. Budgets also help managers develop concrete expectations about the future, thus providing 
standards against which judge subsequent performance. Budgets help coordinate and communicate 
plans and develop congruence between individual and overall organization goals and objectives. 

Budgets measure performance, identify potential problems and, because they give response, permit 
corrective actions. Comparisons of actual performance with planned performance give managers 
attention-directing cues and current readings on performance. These comparisons of actual to planned 
performance are called budget variances. 

Types. Budgets focus on time periods or content. Long-range budgets include the acquisition of major 
new resources such as building and equipment and are called capital budgets. The master budget covers 
an organization's financial operation for an operating cycle, usually one year, and may be on a yearly 
basis or continuous. The continuous or rolling budget is updated in monthly or quarterly increments 
as each month or quarter ends. The rolling budget reflects management's expectations for a year in 
advance, and the most recent operating experiences can be used for setting the budget for the following 
12 months. The continuous budget avoids the once-a-year budget preparation scramble and stabilizes 
planning since new information is added frequently. 

Operational Steps. The master budget consists of 10 steps (Sorensen, et al., 1983). The manner in which 
the steps will become operational depends on the type of human service organizations. The first step, 
a forecast of activity, has a different interpretation by varying health organizations. For example, a state 
human service agency may be budgeting for state appropriations while a county human service agency 
may be projecting revenues from state or county sources (e.g., taxes) and a county or not-for-profit 
provider organization may be forecasting levels of service to clients. 

Step Description 

1. Prepare a yearly forecast of activity by month (or quarter). 

2. Translate the activity forecast into estimates of revenue to be generated. 

3. Estimate any other sources of revenue such as grants or gifts. 

4. Translate all sources of revenue into estimated cash collections by month (or quarter) by 
combining steps 2 and 3. 

5. Estimate the purchases of any services or supplies inventories for each month (quarter). 

6. Estimate the timing and payments for purchases of services and inventories. 

7. Estimate salary and wage expense disbursements and anticipated increases (or decreases) by 
month (quarter). 

8. Prepare the budgeted statement of operations. This statement uses the revenue information 
from step 2 and 3 along with expenses from step 5 and 7. If the organization is a non-profit, 
additional expenses such as depreciation and uncollectible accounts should be added to 
complete the list of expenses. 

9. Prepare the budgeted statement of cash receipts and disbursements. 

10. Prepare the budgeted statement of financial position (or balance sheet) and budgeted statement 
of cash flows (if desired). The projected statement of financial position (or balance sheet) draws on 
the beginning-of-period statement and is adjusted for all the activity described in steps 2 through 9. 
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Many times the resources are insufficient to develop and implement the master budget as described 
in the above 10 steps. Another form of budgeting, known as incremental budgeting, appears in some 
human service organizations. The analysis starts with existing volumes of activity, revenue and expenses 
and then makes incremental adjustments for factors such as changes in the volume, changes in the 
efficiency, changes in the quality and changes in the prices of good and services. Finally after all of 
foregoing adjustments, the revised totals are multiplied by an inflation percentage expressed as a 
multiplier (e.g., 5% = 1.05 multiplier) to produce the new budget. An illustrative format follows: 






New 

Revised 


New 

Last Year 

Volume 

Efficiency 

Quality 

Prices 

Totals 

Inflation 

Budget 


% 

$ 

$ 

$ 

$ 

% 

$ 

[Volume: xxx 





xxx 


xxx] 

Revenue: $xxx 





$ xxx 


$ xxx 

Expense: $xxx 
(or expenditure) 
Over(Under) 





$ xxx 


$ xxx 

Recovery: $xxx 





$ xxx 


$ xxx 


[ ] Brackets indicate non-dollar amounts 

Controlling against the budget. Budgetary control is a comparison of the planned revenues and 
expenses (or expenditures in the case of a governmental unit) to the actual revenues and expenses (or 
expenditures). The difference is a variance. Usually parentheses on a variance indicate an unfavorable 
variance. 

If planned revenue is less than expected, it may be as follows: 


Budget: 

Actual: 

Variance: 

$50,000,000 

$48,000,000 

$(2,000,000) 

If favorable, then: 

$50,000,000 

$51,000,000 

$1,000,000 

Expenses (expenditures) may 

follow the same scheme: 

An unfavorable variance would appear as follows 

$30,000,000 

$31,500,000 

$(1,500,000) 

If favorable, then: 

$30,000,000 

$28,900,000 

$1,100,000 


If the variances follow the logic of parentheses for unfavorable variances, the variances combine 
additively to identify the net effect. In the foregoing example, the two unfavorable variances would 
combine to a negative impact of ($3,500,000) or if the revenues are favorable, $1,000,000, and 
the expenses (expenditures) are unfavorable, ($1,500,000), the net consequence is an unfavorable 
($500,000). 

Direct comparison of the original fixed budget and actual results may be misleading if the actual level 
of activity differs significantly from the planned level of activity. To cope with this issue, a human 
service organization may prepare a flexible budget, which changes as activity levels change. A flexible 


62 


Human Services Program Evaluation 

Financial Management in Human Services Organizations 


Chapter 5 


budget, unlike the adoption of a fixed budget by a governmental unit, is not a form of appropriations, 
but a plan that can facilitate budgetary control and operational evaluations. A flexible budget allows 
the organization to prepare several budgets at different levels of activity to identify an acceptable 
comparative basis of planned activity with actual results. 

Cost accounting (or cost-finding) and rate-setting for services and/or programs 

Costs are associated with some activity, event, situation, product, or service - in short, with some cost 
objective. Costs are associated with "something" and the "something" is a cost objective. If the cost 
objective is the unit cost of services (or cost-per-unit of service), competent cost finding requires ten (10) 
procedures (Sorensen [1], 2000) 

1. Identify and document the organizational units and the services (or programs) of each unit of the 
organization. 

2. Assign the direct salary and wage cost to each organizational unit and to each service (or 
program). 

3. Determine the cost of fringe benefits (e.g., social security, vacation, insurance, education 
leaves) and assign (estimated) fringe benefits to each organizational unit and to each service (or 
program). 

4. Estimate and assign the value of donated services, supplies and facilities (e.g., essential 
volunteers' services or "in kind" expenses) to each organizational unit and to each service (or 
program). 

5. Assign other direct and traceable expenses to each organizational unit and to each service (or 
program). 

6. Assign indirect operating expenses by organizational unit and service (or program). 

7. Assign the costs of administrative and support units to other organizational units and to services 
(or programs). 

8. Determine the most feasible basis for unitizing the services provided by the organization. 

9. Identify the actual (or estimated, if prospective) annual (or some other period) amount of service 
for each service (or program). 

10. Compute the unit cost rate for each service (or program) (step 7 divided by step 9). 

Figure 4 using illustrative data summarizes the 10 cost-finding procedures required to produce unit costs 
for human service services or programs. A federal agency (Substance Abuse and Mental Health Services 
Administration or SAMHSA) funded a uniform system of accounting and cost reporting for substance 
abuse treatment providers (Capital Consulting Corporation, 1993) that embraces the foregoing 
methodology. 
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Figure 4. Schedule of Unit Cost Procedures 


Procedure 1 


Expenses 

Total Costs 

Admin 

Service A 

Service B 

Procedures 2 & 3 

Compensation 

$ XXX, XXX 

$ XX, XXX 

$ XXX, XXX 

$ XXX, XXX 

Procedure 4 

Other direct 
and traceable 

XX, XXX 

XX, XXX 

x,xxx 

x,xxx 

Procedure 5 

Indirect 

XX, XXX 

XX, XXX 

XX, XXX 

XX, XXX 

Procedure 6 

Donated 

XX, XXX 

0 

x,xxx 

0 

TOTAL 

$ XXX, XXX 

$ XX, XXX 

$ XXX, XXX 

$ XXX, XXX 

Procedure 7 

Assigned costs administration 

($xx,xxx) 

+$ XX, XXX 

+$ XX, XXX 

Procedure 8 

Total cost of two major services 

0 

$ XXX, XXX 

$ XXX, XXX 

Procedure 9 

Unit of service 



n,nnn 

n,nnn 

Procedure 10 

Cost per unit of service 


$ XXX 

$ XXX 


Activity-based costing suggests costs should be collected in (1) cost pools of related costs and 
then (2) assigned to specific services based on the usage of the cost pool activity. This two-stage 
allocation procedure can result in improved cost assignments. The costs of the information system 
(e.g., IS personnel, computer, processing costs) may be collected in a cost pool and then assigned to 
organizational units based on utilization (e.g., number of transactions or hours of usage). Administrative 
costs may be collected in a cost pool, for example, and assigned to other organizational units based on 
the number of full time equivalent (FTE) personnel (e.g. two half-time persons equal one FTE) employed 
in each unit (e.g., if a unit had 100 FTE out of a total of 1,000 FTE in the organization, the unit's 
assignment would be 10 % [100/1000 = .10 or 10%]). 

The Center for Substance Abuse Treatment (CSAT) has developed an e-curricula Toolkits for unit of 
service costing and rate setting through the State Systems Technical Assistance Project (SSTAP). To access 
these Toolkits register and access the CSAT SSTA Unit Cost Determination and Rate Setting Toolkits at 
<http://sstap-learn.jbsinternational.com/login/index.php> and create a personal account. By completing 
the assessment in each module, a user can receive continuing professional education (CPE) credit. 

Unit costs are used often for rate-setting, contract negotiation or highlighting problems in rates or 
productivity. Comparison of budgeted unit costs to actual unit costs is a Key Performance Indicator cited 
earlier. Unit costs of comparable organizations can serve as a useful benchmarking process. 

Cost-Outcome and Cost-Effectiveness 

Cost-outcome assessment is one key to building viable cost-effectiveness analyses to perform program 
evaluation and to achieve desired accountability (Newman and Sorensen, 1985). Figure 5 identifies the 
major financial, statistical and evaluation tasks required for cost-outcome and cost-effectiveness analysis. 
[A more complete discussion on cost-outcome and cost-effectiveness was presented in Chapter 4; some 
material is repeated to establish a clear linkage of unit costs with client and program outcomes]. 
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Starting with total costs of a human service organization, costs are refined to the per-unit cost of service. 
Statistical data on professional staff activities are required to assign personnel costs, while information 
about services (e.g., units of service) is necessary to unitize program and service costs. With unitized 
costs of service and accumulated services received by specific target groups, total costs for an episode 
of care may be computed. Evaluation tasks then involve the selection of a target group, pre-intervention 
assessment, and random assignment of clients to varied treatments or services. After post-intervention 
measurements, outcomes are assessed. Then costs can be related to outcomes and, if calculated on 
more than one service, can be comparatively analyzed for assessing the cost-effectiveness of optional 
approaches for specific target groups (Thornton, et al., 1990, Sorensen [2], 2000). 


Figure 5. Overview of Major Tasks in Cost-Outcome and Cost-Effectiveness Studies 
in a Human Service Organization 


Financial Tasks 
(Accounting and Cost 
Accounting) 


Statistical 

Tasks 


Program Evaluation 
Tasks 
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Implications for human service delivery of services 

Financial management focuses on the analyses, decisions and actions related to the acquisition of 
resources (financing), allocation of resources (distribution), and accountability for the use of resources 
(evaluation). 

Strategic financial management. Successful strategic financial management in a human service 
organization links critically to the strategic management process. The future of human service depends 
on how effectively the strategic management process of vision, mission, goals, and objectives is 
understood and applied. Fruitful strategic financial plans depend on assessments of the organization's 
strategic situation, competitive position, strategy formulation, evaluation and choice. These long 
term plans must be enmeshed in a quality orientation that has a focus of ongoing improvement— 
either incremental or breakthrough improvement. The successful strategic processes will focus on the 
customer, quality-driven processes, leadership and management, and engagement and commitment of 
employees. 

Operational financial management. Operational financial management begins with a focus on Key 
Performance Indicators (KPI) drawn from the Information System (IS). The KPI system is a managerial 
guidance system. Fluman service information systems should be shaped to produce ongoing 
comparisons between expected and actual KPI to aid managers in assessing services, how resources are 
acquired and spent, and program outcomes. 

Managers in human service organizations should have a basic understanding cost and revenue 
behavior, how to assess "breakeven" operations, cost classification, differential costs and revenues. The 
fundamentals of the budgeting process, budget variance analysis, and unit of service costing are needed 
to assess meaningfully the financial operations of a human service organization. Finally, costs and client 
(program) outcomes should be linked through cost-outcomes and cost-effectiveness to enhance human 
service program accountability and effectiveness. 

Many human service managers emerge from medical, behavioral and clinical backgrounds and, 
therefore, are not systematically exposed to fundamental strategic and operational financial 
management skills. This chapter, hopefully, begins to meet this need. 

What is the payoff? Managers of human service organizations with effective strategic and operational 
financial management can face an uncertain and threatening environment with greater confidence and 
with higher odds of success. 
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APPENDIX A 

Cost Determination of Substance Abuse Procedures Using a Medicare/ 

Medicaid Framework. 

James E. Sorensen, Ph.D., CPA, CGMA 

August 2014 

Under a Medicare/Medicaid controlled cost reporting and rate-setting environment several new key 
concepts need to be identified. Medicare and Medicaid has adapted an Activity Based Costing (ABC) 
approach known as Time-Driven Activity Based Costing (TDABC) where the cost objects (namely, 
services) are costed based on time (usually physician driven) and a generic average cost of resources 
used to produce the service. The cost-determination and rate-setting turns out to be a non-trivial 
process. While the example is for substance abuse, the logic of the approach is applicable to any service 
paid by Medicare or Medicaid. In many human service agencies, the funder will be Medicaid and, at 
the State level, will involve a Department of Health Care Policy and Financing (namely, the state office 
for Medicare and Medicaid). Often times, the State human service agency will work with the State 
Department of Health Care Policy and Financing in determining the cost and rate for various services 
(now called encounters and procedures). The resulting rates usually affect any provider where Medicaid 
financing is involved and may often require provider participation in the cost determination and rate- 
setting process. An example of how a State became involved in this process is outlined in Mercer (2006). 

Use of Encounters and Procedures 

• Service events are classified as encounters. Encounter data should be used to assess the 
reimbursable services provided under managed care program. Usually encounter data are used 
to set capitated rates for managed care programs in Medicaid where fee-for-service data are 
not available. Encounter data also allow Alcohol and Other Drug Abuse (AODA) to complete 
meaningful analysis and comparison of service utilization and costs and behavioral health 
organizations and various provider groups. 

• An encounter should identify the type of service delivered using standard procedure codes 
required by the federal Health Insurance Portability and Accountability Act of 1996 (HIPAA), the 
recipient and other key detail. 

• Broad groups of encounters are decomposed into procedures. Many states have developed 
lists of procedure code definitions and have created comprehensive procedure data reporting 
manuals. (See Colorado, for example.) These manuals help to direct and monitor providers that 
service delivery is tracked consistently and reported statewide. 

• A specific procedure code is developed for each service. For example alcohol and/or drug 
services for family or couples counseling is assigned a procedure cost of T1006. Generally 
the procedure codes are based on time studies documenting the amount of time required to 
complete the procedure. 

• Encounter data will include ambulatory services (e.g., outpatient, group therapy, supported 
housing) and residential housing. 

• The data in the encounter records should originate at the individual provider level with the 
information forwarded to a behavioral health organization (BHO) and/or to AODA. Some 
providers will deal directly with AODA, thus the "and/or" earlier wording. A key point is all 
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encounters are eventually reported to AODA, either through a BHO or directly by a provider 
working directly with AODA. When encounter data are reported from all providers, the State will 
be able to evaluate the system Statewide. 

Use of Relative Value Units (RVU) 

To differentiate the varied procedures, a relative value weight is assigned to a service (e.g., individual 
psychotherapy, office, 45-50 minutes = 2.30 or family psychotherapy (with patient) = 2.83. These RVUs 
have been established by prior time studies and codified into RVU tables. 

RVUs may be further classified by the location of delivery: facility or non-facility. The location of delivery 
can influence the RVU weight, but not always. For example, a family assessment might be given a facility 
RVU of 2.03 while the same service in a non-facility would be awarded 2.32. 

Encounters and RVU Weights and Non-encounter based substance abuse costs 

Encounters may be further classified by those with RVUs and those without. Costing would be separate 
for (1) encounter-based substance abuse with RVU weights, (2) encounter-based substance abuse 
residential without RVU weights and (3) encounter-based substance abuse without RVU weights and 
non-counter based substance costs. 

Classification of Costs 

Costs are divided between direct costs and indirect costs. 

• Direct costs can be traced directly to a cost center and/or a program (or service). Included are 
costs such as salaries and wages for direct service staff and administrative or operating costs that 
can be directly attributable to a specific program (or service). Direct costs and direct program 
administrative operating costs are jointly classified as direct costs. 

• Indirect costs are those that cannot be directly traced to a program (or service). Indirect costs 
would include general costs such as administration, depreciation, rent (not traced to a specific 
program or service), insurance, etc. 

• Non-encounter based BHO administrative costs. 

• Unallowable costs. 

Assignment of costs to programs and services 

For the sake of clarity and understanding, the cost determination and rate-setting process is divided 
into three Excel spreadsheets. The first uses the step-down method illustrated in the Toolkit for the 
assignment of indirect costs. 

The second Excel spreadsheets identifies all direct costs by (1) encounter-based substance abuse with 
RVU weights, (2) encounter-based substance abuse residential without RVU weights and (3) encounter- 
based substance abuse without RVU weights and Non-counter based substance costs. 

The third Excel spreadsheets combines the results from the prior two Excel spreadsheets to identify total 
costs. 
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RVU and Base Unit Costs (BSU) and Resulting Rate for a Procedure 

The RVU are accumulated for all procedures provided to develop a total RVU for all encounter-based 
procedures. After the costs of these same services is determined (see above), the total cost is divided by 
the total RVU to develop a Base Unit Cost (BUC). The BUC is then applied to the RVU to determine the 
rate for a given procedure. 

Other services such as a residential service (encounter with no RVU) may be costed by dividing by, say, 
residential days of care. 

Comprehensive Example 

Seven templates explore the entire costing process using the Medicare and Medicaid framework. The 
illustration is based on a $5,000,000 organization with $1,000,000 of indirect costs and $4,000,000 of 
direct costs. The same process would apply to smaller or larger operations. The values illustrated are to 
make the example easy to follow, but are not intended to represent a typical size. 

Figure 1 explores the INDRECT cost allocation to the various encounter categories. An accounting manual 
would be necessary to specify how various indirect costs would be allocated to the encounter categories. 

Figure 2 explores the DIRECT cost assignment to the various encounter categories. Presumably the 
accounting system would accumulate the direct costs as incurred by the programs (services). 

Figure 3 combines the indirect and direct costs into a single document. In this illustration, the Encounter- 
Based Substance with RVU weights totals $4,290,000 (namely, direct of $3,390,000 and indirect of 
$900,000). 

Figure 4 illustrates the accumulation of Relative Value Units (RVUs) using two illustrative services. This 
schedule would be normally an extensive listing of procedures available in an encounter environment. 
The Total Facility based RVUs is (an assumed value) 61,285. 

Figure 5 computes the Base Unit Cost (BUC). The Total Allowable costs for Encounter-Based events are 
$4,290,000 and the Total Relative Value units (assumed) is 61,285 for a calculated BUC of $70. 

Figure 6 calculates the cost per unit of service (procedure) by multiplying the RVU (facility or non-facility) 
times the Base Unit Cost (BUC). In this example, procedure 90807 with a facility RVU of 2.5 (Individual 
Psychotherapy, Office 45-50 minutes) is multiplied by $70 for a cost of $175.00. This process would be 
repeated for all procedures for both facility and non-facility settings. 

In the absence of other factors, this $175 would be billing rate for Medicare/Medicaid services. Other 
factors might include specific adjustments for the circumstances of the provider. 

Figure 6 demonstrates how a Non-Encounter Value Unit Residential Day would be determined. The total 
costs is divided by the total number of days or $680,000/7158 = $95. 

The cost per day would also service as the billing and payment rate for Medicaid/Medicaid payments. 
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Reference: 

Mercer Government Human Service Consulting. (2006). Medicaid Mental Health Rates. Department of Health Care Policy 
and Financing. Performance Audit. (An electronic version of this report is available on the Web site of the Office of the State 
Auditor <http://www.state.co.us/auditor>. The report control number is 1754). 
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Figure A-6. Template for the Computation of Cost Per Unit of Service with RVUs 
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Difficulties encountered in implementing effective 
program evaluation systems. 


Implementing an effective program evaluation system always requires a considerable amount of creative 
problem solving. No two organizations are exactly alike. No two funding systems are exactly the same. 
The administrative structure and organizational charts are always somewhat different. Certainly the 
personalities for the individuals within an organization vary tremendously. If a program evaluation 
system is to succeed, it must be flexible enough to address the unique issues within the organization it is 
intending to impact. In particular, if the program evaluation is to have an impact on the decision-making 
within an organization, the characteristics of the decision-making process must be addressed. 

This chapter will provide some suggestions for how some common problems can be addressed. These 
suggestions are intended to be illustrative of the types of problem solving that must happen if the 
program evaluation system is going to work. 

The issues in this chapter are arbitrarily divided into issues internal to the program or organization and 
issues external to the organization. The internal issues include implementation problems encountered 
such as staff resistance and politics within the organization. The external issues include issues associated 
with reporting information to funders or regulatory requirements imposed by external funding or 
regulatory agencies. Traditionally, public or quasi-public agencies have had to be more concerned with 
reporting requirements because much of their funding came from government sources. However, in 
today's environment the line between public and private agencies has blurred considerably. Private 
hospitals in the United States often receive a considerable amount of their funding from Medicaid 
and Medicare. Public agencies like community mental health centers often have contracts with private 
insurance programs and are sometimes designated managed care providers. In the international 
community non-governmental organization (NGOs) provide much of the behavioral health services. Their 
funding comes mostly from donors but they often partner with governments in low and middle income 
countries. Consequently, they may operate like quasi-governmental organizations in the US. All human 
service programs have some formal or informal accountability requirements that are intended to show 
that the organization is doing what it is supposed to be doing. 

What are the typical internal problems a program evaluator encounters when 
developing a program evaluation system? 

The two basic problems are: 

a. Resistance from the line staff (direct service providers) who consider themselves to be the 
primary focus of the evaluation effort. 

b. Misunderstanding on the part of the administrative and management staff regarding the 
potential benefits of a program evaluation system. 

Typically, line staff are most concerned about the potential impact of a program evaluation system. Even 
though program evaluation evaluates the extent to which the services provided by the organization as 
a whole are effective and efficient, line staff rarely understand and accept this premise. Line staff often 
fear that the results of the program evaluation will create some problems, including: 
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1. The results of the evaluation will reflect negatively on their work 

2. The evaluation results will make them change the way they do their work. 

3. The evaluation system will make the services impersonal and mechanical. 

4. The evaluation system will focus on things that are not really important. 

5. The evaluation system will require that much more time and energy be devoted to surveys 
and reports and the result will be reduced services. 

Effective resolutions to these issues must involve both education and ongoing dialogue between the 
service provider and the individual(s) responsible for the evaluation. Some basic principles for evaluation 
designs that minimize service provider resistance are: 

1. Make the evaluation system design public. 

2. Solicit input from service providers and incorporate good suggestions into the design. 

3. Develop practical applications for the evaluation system. 

4. Minimize the amount of additional data required of clinicians. 

As in any system change project the amount of investment of the line staff in the project is directly 
related the extent to which they feel some sense of ownership. Line staff can be expected to be resistive 
if the system is imposed from some outside source or (even worse) from internal management, On 
the other hand, if line staff is an active participant in the design and implementation of the project and 
expects to benefit in some way, enthusiastic participation is more likely. 

What kinds of issues are administrators and managers likely to have with a program 
evaluation system? 

Administrators may have some of the same kind of concerns as line staff. In particular, administrators are 
often concerned that the program evaluation will reflect negatively on the organization and indirectly on 
the administration. Also, administrators may be concerned that the program evaluation system may be 
more costly and time-consuming than will be justified. However, since administrators and managers are 
typically more involved in the decision to do program evaluation, many of these concerned have been 
addressed before a program evaluation effort is initiated. Nevertheless managers and administrators 
often have unrealistic expectations about what program evaluation can and cannot do. 

Sometimes, administrators and managers have had to convince themselves, a board of directors, or 
others within the management group that a program evaluation effort would be beneficial. This process 
can be very useful for identifying potential benefits and potential problems. However, some people may 
come away from this process with the expectation that program evaluation can answer and address 
any management issue. (Tom has worked with several organizations whose administrators were initially 
resistive to program evaluation efforts. However, after they learned more about the potential benefits, 
they became enthusiastic supporters. And they sometimes expected that program evaluation could solve 
all the management problems.) Program evaluation and information from this system can provide much 
useful guidance in making decisions. Rarely is the information so unambiguous that everyone agrees on 
the direction to be taken. Some amount of judgment and leadership is a necessary ingredient in ensuring 
that program evaluation and management expertise will combine into a beneficial partnership. 

These are some of the principles that will help to ensure that administrative issues are minimized: 
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1. Establish reasonable expectations and timetables. 

2. Discuss possible scenarios beforehand so that there are fewer surprises 

3. Produce some results quickly and emphasize the positive aspects of the information. 

What are the external reporting requirements and what are the problems associated 
with these requirements and program evaluation? 

Regulatory and funding agencies have typically focused their oversight on compliance with a 
predetermined set of regulations. These regulations vary widely from state to state and from agency 
to agency. However, almost all existing regulations are based on assumptions about what activities 
are associated with quality care. Few of these regulations are the result of good research that links 
these activities with good consumer outcomes. The more common regulations include requirements 
about what is to be included in clinical charts, what is to be included in various policy and procedural 
documents, and how clearly responsibilities are outlined. 

Fortunately, there is an effort within many of these agencies to identify and evaluate the extent to 
which desirable consumer outcomes have been achieved. SAMHSA (Substance Abuse and Mental 
Health Services Administration) has been very active in developing a set of outcome measures that 
would be required in all of their SAMHSA funded programs. (Tom was involved in the early stages of this 
development when he was the president of the National Association of State Mental Health Program 
Directors (NASMHPD) board.) The development of a common set of reporting measures has obvious 
benefits for organizations that have multiple funding sources with unique reporting requirements. 

The lack of agreement on reporting measures can create problems for provider organizations since 
compliance with regulatory and accrediting bodies is necessary for most organizations to continue 
receiving funding. Conflicting requirements on outcomes could force provider organizations to collect 
duplicative and/or overlapping information. For example, regulatory agencies often require consumer 
satisfaction information. If the information requested is different across different reporting agencies, the 
provider organization is required to collect duplicative consumer satisfaction information. 

Also, if an organization decides to generate consumer outcome information in addition to the 
information necessary for regulatory and accrediting bodies, there is some risk involved. If the 
information collected is not part of the consumer outcome information that will be required in the 
future, the organization will either have to scrap its internal outcome data system or agree to collect the 
new information in addition to the internal information. 

The most acceptable solution to this dilemma is for organizations to develop consumer outcome 
information that is valuable regardless of what information may be required from other organizations 
in the future. This information should include outcome information that can be used to improve clinical 
programs and the clinical decision-making process. 

In addition, the outcome information that is required by external funding, regulatory, or accrediting 
bodies should be utilized to provide useful information internally. As an example, when Tom was mental 
health director for Colorado, the contract agencies were required to collect and submit the following 
information (Barrett et al, 1998): 
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1. Consumer satisfaction information. 

2. Level of functioning information on all consumers admitted and discharged from the public 
mental health system. 

3. Employment information on admission and discharge 

4. Living status information on admission and discharge. 

Information from these measures was utilized to report to the Colorado legislature to demonstrate the 
benefits of providing mental health services to the citizens of Colorado. Also, the information could be 
used to provide feedback to clinical staff on performance, to evaluate organizational performance and to 
improve the outcomes from clinical programs. 

What measures are being considered for use across organizations? 

There have been several national efforts to develop a core set of outcome indicators that can be used 
across all publicly funded mental health agencies. In 1997 the Mental Health Statistics Improvement 
Program (MHSIP) proposed some standardized scales to evaluate mental health programs (Teague et al., 
1997). More recently SAMHSA developed the National Outcome Measures (NOMs) (SAMHSA website, 
accessed August 12, 2014). The indicators included in these two projects are: 

1. MHSIP report card survey (Mental Health Statistics Improvement Project) 

• Increase access to general healthcare. 

• Minimize negative outcome for treatment. 

• Reduce psychological distress. 

• Increase sense of personhood. 

• Decrease impairment from substance abuse. 

• Increase productive activity. 

• Capacity for independent community living. 

• Reduced involvement in criminal justice system. 

• Minimal recurrence of problem. 

• Positive changes in areas for which treatment is sought. 

• Increase natural supports and social system. 

2. Original reporting requirements under NOMs were: 

• Reduced Morbidity. 

• Employment/Education. 

• Crime and Criminal Justice. 

• Stability in Housing. 

• Social Connectedness. 

• Access/Capacity. 

• Retention. 

• Perception of Care (or services). 

• Cost Effectiveness. 

• Use of Evidence-Based Practices. 

Since all SAMHSA funded programs are required to submit URS data, US government funded programs 
should consider utilizing these measures as part of their ongoing data collection systems. 
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At the international level, the World Health Organization Assessment Instrument for Mental Health 
Systems (WHO-AIMS) (Saxena et al, 2007) identifies some basic dimensions for evaluating mental health 
systems in low and middle income countries: 

1. Policy and Legislative Framework. 

2. Mental Health Services. 

3. Mental Health in Primary Health Care. 

4. Human Resources. 

5. Public Education and Links with Other Sectors. 

6. Monitoring and Evaluation. 

This instrument has 155 items that are intended to evaluate the quality and maturity of mental health 
systems in countries. The instrument is very comprehensive and can be used to evaluate developing 
mental health systems 

Useful resource materials 

The Evaluation Center@HSRI is a national technical assistance center for the evaluation of adult mental 
health systems change. The mission of the Evaluation Center@HSRI is to provide technical assistance 
in the area of evaluation to States and nonprofit public entities within the States for improving the 
planning, development, and operation of adult mental health services carried out as part of the 
Community Mental Health Services Block Grant program. The Evaluation Center@HSRI is a technical 
assistance center funded by the federal Substance Abuse and Mental Health Services Administration 
(SAMHSA), Center for Mental Health Services (CMHS) and operated by the Human Services Research 
Institute (HSRI). 
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Useful resource links: 

Behavioral Health 

• Center for Mental Health Services (CMHS) 

• Center for Psychiatric Rehabilitation (CPR) 

• Center for Substance Abuse Treatment (CSAT) 

• Department of Health and Human Services Health Resources and Services Administration 

• National Association of State Mental Health Program Directors (NASMHPD) 

• National Council for Community Behavioral Healthcare (NCCBH) 

• National Institute of Mental Health (NIMH) 

• Research and Training Center for Children's Mental Health, Florida Mental Health Institute 

• Substance Abuse and Mental Health Services Administration (SAMHSA) 

• Western Interstate Commission for Higher Education (WICHE) 

Consumer, Family & Advocacy 

• Mental Health America and Mental Health America of Colorado 

• National Disability Rights Network 

• National Alliance for the Mentally III (NAMI) 

• National Association for Rights Protection and Advocacy (NARPA) 

• National Empowerment Center, Inc. (NEC) 

• National Mental Health Consumers' Self-Help Clearinghouse 

• National Technical Assistance Center for Children's Mental Health 

Developmental Disabilities 

• National Association of Councils on Developmental Disabilities 

• National Association of County Behavioral Health Directors (NACBHD) 

• The National Association of State Directors of Developmental Disabilities Services (NASDDDS) 

National Quality Assurance/Measurement Systems 

• National Committee for Quality Assurance (NCQA) 

Multicultural Related Resources 

• Asian & Pacific Islander American Health Forum (APIAHF) 

• Diversity RX 

• Electronic Magazine of Multicultural Education (EMME) 

EMME is an on-line magazine for scholars, practitioners and students of multicultural education. 
Uniquely theme-oriented, each issue of EMME contains articles, instructional ideas, and reviews 
of juvenile and professional books and multimedia materials on a particular topic. EMME is made 
available to the general public only in the electronic format and without any subscription fee. 

• Ethnomed 

The EthnoMed site contains information about cultural beliefs, medical issues and other related 
issues pertinent to the health care of recent immigrants. 

• Gay, Lesbian, Bisexual, Transgender (GLBT) Health Access Project 

The mission of this organization is to strengthen the Massachusetts Department of Public 
Health's ability to foster the development of comprehensive, culturally appropriate health 
promotion policies and health care services for GLBT people through a variety of venues 
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including community awareness, policy development, advocacy, direct service and prevention 
strategies. 

• National Center for American Indian and Alaska Native Mental Health Research (NCAIANMHR) 
The NCAIANMHR, a program in the Department of Psychiatry at the University of Colorado, is one 
of four minorities mental health research Centers. The NCAIANMHR is sponsored by the National 
Institute of Mental Health and is the only program of this type in the country focusing specifically 
on American Indian and Alaska Native populations. 

• National Center for Cultural Competence (NCCC) 

Major emphasis is placed on policy development, assistance in conducting cultural competence 
organizational self-assessments, and strategic approaches to the systematic incorporation of 
culturally competent values, policy, structures and practices within organizations. 

• National Center for American Indian and Alaska Mental Health Research 

• National Indian Child Welfare Association (NICWA) 

• National Institute of Mental Health/Office of Rural Mental Health Resources (ORMHR) 

Directs, plans, coordinates, and supports research activities and information dissemination on 
conditions unique to those living in rural areas, including research on the delivery of mental 
health services in such areas; and coordinates related Departmental research activities and 
related activities of public and nonprofit entities. 

• National Research Center on Asian American Mental Health (NRCAAMH) 

• National Rural Health Association (NRHA) 

The National Rural Health Association is a national membership organization whose mission is to 
improve the health and healthcare of rural Americans and to provide leadership on rural issues 
through advocacy, communications, education and research. 

• Program for Research on Black Americans (PRBA) 

The Program for Research on Black Americans is a research group at the University of Michigan's 
acclaimed Institute for Social Research. PRBA's goal is to produce high quality national survey 
research on Black Americans. 

• The Center for Cross Cultural Research 

• The Center for Research on Ethnicity, Culture and Health (CRECH) 

The Center for Research on Ethnicity, Culture and Health, established in 1998, provides a 
forum for basic and applied public health research on relationships among ethnicity, culture, 
socioeconomic status and health. 

• The Minority Health Network (MHNet) 

• The Minority HIV/AIDS Initiative 

The Minority HIV/AIDS Initiative provides funds to community-based organizations, faith 
communities, research institutions, minority-serving colleges and universities, health care 
organizations, state and local health departments, and correctional institutions to help them 
address the HIV/AIDS epidemic within the minority populations they serve. 

• The National Alliance for Hispanic Health 

The National Alliance for Hispanic Health is the oldest and largest network of health and human 
service providers servicing over 10 million Hispanic consumers throughout the U.S. Since 1973 
we have grown from a small coalition of visionary mental health providers to a large, dynamic, 
and strong group of organizations and individuals. 

• The Office of Minority Health Resource Center (OMHRC) 

The mission of OMH is to improve the health of racial and ethnic populations through the 
development of effective health policies and programs that help to eliminate disparities in health. 
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Outcome Measurement 

• Adult Mental Health Workgroup 

The Adult Mental Health Workgroup was formed as an outgrowth of the Consensus Forum on 
Mental Health and Substance Abuse Performance Measures at the Carter Center in the spring 
of 2001. Leaders representing a wide range of concerned groups came together to assess the 
progress made to date on the development and implementation of performance measures in 
behavioral health care and related service systems. 

• Center for the Study of Issues in Public Mental Health, The Nathan Kline Institute 

• Outcome Measurement Resource Network 

• PacifiCare ALERT outcomes management system 

Annual Report: presents graphical information and explanatory text about results of various 
outcome and quality measures 

• National Association of State Mental Health Program Directors 

• National Research Institute (NRI) (the research arm of the National Association of State Mental 
Program Directors) 

Research, Policy & Practice 

• Association for Health Services Research 

• Mental Health Recovery Research 

• National GAINS Center 

• National Association of State Mental Health Program Directors 

• National Research Institute (NRI) (the research arm of the National Association of State Mental 
Program Directors) 

Research Resources and Technology 

• FedWorld Information Network 

• Internet Mental Health 

• Internet Resources for Nonprofits 

• Mental Health InfoSource 

• Mental Health Net 

Rural Mental Health and Homelessness 

• Frontier Mental Health Services Resources Network 

• Health Care for Homeless Information Resource Center 

• National Resource Center on Homelessness and Mental Illness 

• The National Coalition on the Homeless (NCH) 

Substance Abuse 

• Center for Education and Drug Abuse Research (CEDAR) 

• Center of Psychosocial and Forensic Services Research 

• Dual Diagnosis Web Site 

• National Clearinghouse for Alcohol and Drug Information (NCADI) 

• Policy Information Exchange 
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Research designs help the evaluator obtain answers to research questions. Basically, research designs 
provide the investigator with a way to study relationships between the services provided and treatment 
outcome. An evaluator should choose a research design that answers questions as accurately and 
economically as possible. In addition to choosing a suitable research design, an evaluator must be aware 
of the sources of biases that may threaten the validity of research findings. Before reviewing different 
research designs, we will discuss the various contaminants of research findings. 

Bias in research 

There are many factors that can "contaminate" research findings. Contamination refers to situations in 
which factors other than the intervention are responsible for the results. Those other factors are threats 
to internal validity (Kazdin, 1992). The following is a listing of the sources of internal invalidity that occur 
most often. 

1. History: Contaminating history can be any event that takes place during the treatment phase 
unrelated to the treatment that may account for the particular outcome. History refers to effects 
of daily events on either individuals or groups. History for individuals includes events such as 
getting a promotion at work, conflict between staff members, or receiving advice from friends. 
Group events can include advertising campaigns and media reported events such as research 
findings. The influence of these events can influence performance on outcome measures. It is 
important to be able to differentiate between the effects of particular circumstances from the 
effect of treatment intervention. 

2. Maturation: Maturation reflects general changes in client(s) that are not specific to the 
treatment interventions (e.g., getting older, growing stronger, becoming smarter, etc.). For 
example, in studies of teaching methods, maturation or "readiness" of young students could 
affect performance on reading tests regardless of whether or not the teaching methods worked. 
Historical events and maturation can occur concurrently. 

3. Practice Effects: Performance on a post-test may be influenced by the client's familiarity with 
the test because he/she learned something about the test during the pre-test. Therefore, 
better performance on the post-test may be attributable to repeated testing rather than the 
introduction of treatment. 

4. Instrument Change: This refers to the variances that occur in the measuring instrument or 
measuring procedures over time. Such changes in instrumentation include administration of a 
new version of the test, and different staff providing subtle variations on the instructions. 

5. Statistical regression: A statistical phenomenon that occurs when an instrument is re- 
administered (e.g., post-test) to subjects or clients with extreme scores. Under these 
circumstances, subjects' scores will tend to regress toward the mean score of the distribution. 

For example, if subjects with scores above a certain score on a pre-test are selected to be 
involved in a study, it is likely that the average scores will be closer to the mean on the post-test. 

6. Selection bias: Selection bias occurs when clients are "selected" or assigned by staff to receive 
a particular form of treatment or intervention. If clients are selected for the intervention, then 
the results may be skewed because of this selection. For example, a clinician selects clients to 
receive a certain treatment(s) based on the severity of diagnoses; the results may be increased or 
decreased because of the selection criteria. 

7. Attrition: Clients who drop out of treatment, regardless of the reason, may influence the 
outcome results. Moreover, clients who terminate treatment are likely to differ from those 
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clients who chose to remain in treatment. Clients who drop out may vary on numerous variables 
including the duration, frequency, and intensity of their disorder. If a disproportionate number 
of clients drop out, then the results may only be valid for the group of clients that completed the 
program. 

8. Interactive effects: Treatment outcome can be influenced by the interaction between the 

treatment method and the client's characteristics (e.g., age or gender) or between the treatment 
method and the pre-test. 

Symbolic notation of research designs 

To describe research designs, Campbell & Stanley (1963) developed notations to diagram designs. The 
symbols used to denote designs include the following: 

R = random assignment of clients to specific conditions 
Non R = non-random assignment of clients 
O = the observation or assessment of performance 
X = the experimental maneuver or treatment intervention 

Experimental Designs 

An experimental design is a scientific design in which the researcher manipulates at least one treatment 
and records changes in the outcomes. An experimental design offers the best way to control for 
extraneous variables. For example, a control group design helps to identify if the treatment intervention 
was responsible for outcome or if some extraneous variable was responsible. Experimental designs can 
also allow numerous hypotheses to be tested at once. 

• Control group design consists of two or more groups. The clients are randomly assigned to the 
groups. One group receives the treatment intervention and the control group does not. Clients in 
both groups receive a pre-test and a post-test. The effect of treatment is the measure of change 
from pre- to post- intervention assessment. This design control for many potential contaminants 
including history, maturation, practice effects, instrument change, and statistical regression. It is 
diagrammed as follows: 


• Solomon four-group design controls for the interaction effects of a pretest and treatment 
intervention. Although the test has strong methodological properties, this design is not 
widely chosen because of the monetary expense (Kazdin, 1992). It is essentially two sets of 
simultaneous experiments which means that research time is doubled, and presumably, so is the 
cost. The Solomon four-group test is diagrammed as: 


R 

R 


01 

03 


X 02 (treatment group) 
04 (control group) 


R 

R 

R 
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01 X 
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X 


02 
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06 
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• Factorial designs enable the researcher to manipulate two or more variables at the same time. 

It allows for the study of interactive effects of the treatment or intervention method(s) on the 
outcome. In its purest form, there are two variables with two distinct levels of conditions, noted 
as 2 x 2. An example of a 2 x 2 factorial design is given: 

Treatments 


Client's Gender 

A 

B 


Treatment A 

Treatment B 

Male 

with male client 

with male client 

Female 

Treatment A 

Treatment B 


with female client 

with female client 


Quasi-Experimental Designs 

Quasi-experimental designs are a type of design in which the conditions of a true experiment can only 
be approximated (Herman, Morris, Fitz-Gibbon, 1987). Specifically, one or more of the requirements for 
a true experiment are not met (Kazdin, 1992). For example, clients are not randomly assigned to receive 
certain interventions. Undoubtedly, this particular constraint will weaken the viability of the research 
conclusions. Examples of quasi-experimental designs are time series and nonequivalent control group. 

• Time series design (also known as AB design) incorporates a specific intervention following 
several observations. The design can demonstrate the effects of an intervention by establishing 
several baselines followed by an intervention, and then observation. The design can also provide 
evidence for the long-term effects of treatment. 

non R 01 02 03 X 04 05 


CHANGE 

TIME 

• Nonequivalent control group design permits clients to receive pre- and post measures at 
different points in time. 

non R 01 X 02 

non R 03 04 


Non-Experimental Designs 

Non-experimental designs are ones in which the researcher cannot manipulate variables or assign 
subjects or treatments at random (Kerlinger, 1986). In program evaluation, non-experimental research 
designs are often used because the very nature of the situation does not permit for random assignment 
or accurate manipulation. The main limitation of non-experimental designs is the increased risk of 
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improper interpretation of the data (e.g., post hoc fallacy). The most commonly found non-experimental 
designs are post-test only, pretest-posttest, and static group comparison. 

• Posttest only design (also known as B-design or "one-shot case study") consists of providing an 
intervention and then measuring the outcome with a post-test. 

non R X 01 

• Pretest-posttest design consists of administering a pre-test, followed by an intervention, and then 
post-test measurement. 

non R 01 X 02 

• Static group comparison design consists of at least two groups with the absence of a pre-test. 

The effects of the intervention are assessed on a post-measure basis only. 

non R X 01 

non R 02 


Summary 

The purpose of a research design is to answer research questions and control extraneous variables 
(Bickman. & Rog, 1997). Extraneous variables can decrease the validity of research conclusions. The 
various threats to internal validity include history, maturation, practice effects, instrument change, 
statistical regression, selection bias, attrition (drop-outs), and interactive effects. 

The investigator's task is to choose an appropriate design to answer the question in a manner that is 
feasible in the setting (Tashakkori & Teddlie, 2002). Different types of research designs are appropriate 
for different settings. The investigator should choose from among the following designs: Control group, 
Solomon four-group, and factorial should be the first choice; Quasi-experimental designs (e.g., time- 
series and non-equivalent control group) should be the second choice; and, non-experimental designs 
(e.g., posttest only, pretest- post-test, and static group comparison) should be the third choice. 
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Implementing and maintaining the program evaluation is a collaborative process. Establishing a 
collaborative working relationship with the organization's personnel will aid in the evaluation process. 
One way to ensure a smooth implementation is to make sure that all levels of the organization 
understand the evaluation goals and plan. 

The following information should be made available to all staff: 

1. The steps involved in the evaluation process (i.e., development of goals and objectives, needs 
assessment, outcome measures, etc.). 

2. Who is responsible for which action steps. 

3. How line staff/therapists, managers, and administrators will be involved in the design and 
implementation. 

A program evaluation steering committee is a good idea in many situations. The steering committee 
should include both program evaluators and representatives from all levels of the organizations. The 
steering committee can provide a link between the staff with the organization and technical evaluators. 
This committee could have several functions: 

1. Disseminate information about the evaluation design and plan. 

2. Identify staff concerns about the evaluation plan. 

3. Resolve data collection issues. 

4. Disseminate information to the staff, administration, the 
general public, and funders about the evaluation results. 

A steering committee can also be useful when an evaluation 
is sponsored by several different organizations. For example, 
international projects often include funding from governmental 
organizations (Grand Challenges Canada), private foundations 
(e.g.. the Carter Center), and other funding sources (e.g. private 
contributions). In these situations, some coordinating body is 
critical for a successful implementation. 

The following are some key strategies for implementation and 
maintenance of the evaluation effort: 

1. Keep the program evaluation system flexible. 

2. Keep information simple. 

3. Keep information "audience specific". 

4. Report information to people who are in a position to 
impact change - therapists. 

5. Ensure interface between program evaluation and 
decision-makers - management. 

The evaluation process must be flexible in order to keep pace with 
the rapid changes in today's health care environment. There is 
little utility in collecting information that is irrelevant to current 


Program Evaluation Workflow 
Diagram 
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goals and directions. The Figure shows how program evaluation works as a feedback process, and 
how the results of the evaluation can suggest adjustments in the organization's goals and objectives 
(Wheelen & Hunger, 2000). [The feedback process was introduced first in Chapter 5, Figure 1 in viewing 
the strategic management process.] 

In presenting the results, the evaluator needs to consider the audience. The information should be 
understandable and should have utility for individuals in the audience. The evaluator should consider the 
following issues in preparing information for presentations and reports: 

1. How will the information be presented? 

2. What information will be presented? 

3. How much detail is necessary to include for the given audience? 

4. How will future results be presented? Who will present the results? 

5. How often are formal reports needed? 

Most importantly, the conclusions from the evaluation should be presented to individuals who are in a 
position to make changes. This means that results must be shared with the line staff providers\therapists 
as well as administrators. It is at this point quality improvement can be formulated and then deployed 
throughout the organization (Evans & Lindsay, 2002), 


Summary 

The success of maintaining and implementing the evaluation process is dependent on the staff and 
administrators within all the involved organizations understanding why evaluation is important, and why 
their involvement is needed. 

The evaluator should maintain a collaborative working-relationship with all levels of the organization. 

In reporting the results of the evaluation, the evaluator should consider: Who will be receiving the 
information? What information is important to disclose? How will the information be presented? The 
evaluator in conjunction with the steering committee and/or the organizational staff should devise a plan 
to monitor and maintain the evaluation process. 

The evaluation systems should be implemented and maintained with a high degree of collaboration 
within the organizations. In fact, evaluation systems work best when the organizations have embraced 
the evaluation systems and are motivated to utilize the system to improve service delivery. 
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The evaluator has many roles within the evaluation process. Some of these roles, such as planner, 
scientist, and advisor, are an inherent part of being an evaluator. Other roles, however, are determined 
by the organizational preferences and organizational structure. For example, an evaluator may be 
"internal" or "external" to the organization. Additionally, an evaluator may be part of the administrative 
decision-making group for the organization. On the other hand, an evaluator may be a separate entity 
of the organization that does not have impact on the decision-making process. It is important that the 
evaluator understands his/her role within the organization at the onset of the program evaluation. 

Internal evaluator 

It is not uncommon for organizations to choose an evaluator that is an employee of the organization. 
Organizations may opt for an internal evaluator for several reasons: 

1. The accessibility of the individual 

2. The individual's familiarity with the organization 

3. The financial constraints of the organization. 

Since the "internal" evaluator tends to have a vested interest in the results of the evaluation, there exists 
a potential risk to the objectivity of the evaluation. 

In today's environments, the function of program evaluation is often performed by several people within 
an organization. Information specialists, quality improvement personnel, and administrators often 
assume the responsibility for program evaluation in addition to other responsibilities. In these instances, 
coordination of functions is especially critical. Also, it is important that the leadership aspect of program 
evaluation not be lost when several people are responsible for program evaluation. 

External evaluator 

A program evaluator that is external to the organization, such as a consultant, may offer a "fresh" 
perspective to the organization. External evaluators may be preferred when there is a need for 
evaluation of a time-limited project, when the credibility of an outside program evaluator is important, 
or when there is a need for evaluation that cannot be influenced by internal "politics." As well, the 
external evaluator may be more readily accepted by staff as an "expert". 

The limitations of an external evaluation include an inability of the evaluator to ensure that the 
recommendations are implemented and limited ability to structure the evaluation in order to address 
day to day decision-making. 

Which is better? Internal or External? 

When choosing an evaluator, an organization should consider the size of the organization, financial 
resources, and the staffing configuration. Additionally, the organization should assess the benefits 
and limitations of internal and external evaluators. The following figure outlines the advantages and 
disadvantages of both. 


Advantages 


Disadvantages 


Internal evaluator 

(e.g., employee of 
organization) 


* lower cost 

* knowledge of organization 

* easier to follow-up on issues 


* potential for decreased objectivity 

* may be perceived by staff as biased 

* susceptible to politics of organization 
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Advantages 


Disadvantages 


External evaluator 

(e.g., consultant) 


* less susceptible to politics 


* higher cost 


* perception of "expert from afar" * not as familiar with organization 

* may provide a new perspective 


Regardless of whether the evaluator is "internal" or "external," there are several functions that must be 
addressed to ensure that the evaluation is successful. First, the evaluation must be organized so that 
there is attention to all phases of the evaluation: Development, implementation, and maintenance. 
Someone within the organization must take on the responsibility of ensuring that the evaluation stays 
"on track." 

Second, during the course of the evaluation there will be many issues that must be resolved. The 
evaluator or evaluators need to assume responsibility for resolving these issues in a manner that 
maintains the integrity of the evaluation, and also ensure the continued support of staff and the 
organization. 

• It will be easier to address staff resistance if the responsibility for data collection is spread 
throughout the organization and is not solely the responsibility of the service providers. 

• Timely and accurate feedback reports may help lower resistance as well. Feedback to the service 
providers can improve the service provider's understanding of an evaluation strategy and can 
stimulate a higher degree of willingness to participate in the data collection effort. 

• Evaluators should realize that it may not be necessary to require a continuous information input 
about all aspects of all programs. In some cases an episodic collection at selected times can 
produce effective evaluations and may lower staff resistance toward participation. 

• The areas to be evaluated should be chosen carefully. In selecting areas to be assessed, 
wide-spread participation by those who will be evaluated tends to foster ownership and buy-in 
as well as the selection of appropriate and relevant measures. 

• Usually there is natural roll-up of indicators from the service provider, to team managers, to 
program directors, to center executives, to the board of trustees, and for external reporting 
to funding agencies. The multi-layer levels of interest should be specified with appropriate 
safeguards for confidentiality so everyone can understand who gets to see what and when and 
in how much detail. 

Of course, evaluators also often assume the roles of negotiators and problem-solvers as well as 
consultants and advisors to the organization. 

Melissa Conley-Tyler (Conley-Tyler, 2005) discusses the importance of making a decision about using an 
interval versus external evaluation. She presents a table to utilize making the decision about which type 
of evaluation is best under which circumstances. 
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